shell


wget downloads files from FTP site TWICE


I'm downloading files under a folder on a FTP site using
wget -A gz -rc -np -nd -l1 -t0 ftp://path/to/folder/
wget downloads index.html first (and then parses urls from it?) and then rejects it.
I checked the links in index.html. For each file, there are 3 links to it:
href=file.gz
href=file.gz;type=a
href=file.gz;type=i
The file at ftp://path/to/folder/file.gz is downloaded and saved to file.gz. Then the file at ftp://path/to/folder/file.gz;type=a is also downloaded and saved to the same file (a redundant download).
Since type=i in defaults, the file at ftp://path/to/folder/file.gz;type=i is the same as the first one and no file transferring actually happens.
How can I avoid the downloading of file.gz;type=a? I tried to add option -R "type=a" but it doesn't work.
Here is an example:
$ wget -A "C*.dataless" -rc -np -nd -l1 -P "/tmp/" ftp://ds.iris.edu/pub/userdata/Ed_Anderson/
--2017-05-07 01:55:46-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/
Proxy request sent, awaiting response... 200 OK
Length: 5522 (5.4K) [text/html]
Saving to: ‘/tmp/index.html’
index.html 100%[============================================================>] 5.39K --.-KB/s in 0s
2017-05-07 01:55:49 (178 MB/s) - ‘/tmp/index.html’ saved [5522/5522]
Removing /tmp/index.html since it should be rejected.
--2017-05-07 01:55:49-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless
Proxy request sent, awaiting response... 200 Gatewaying
Length: 65536 (64K) [text/plain]
Saving to: ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’
COCO.IS.EpiSensor.100hz.860384.da 100%[============================================================>] 64.00K 209KB/s in 0.3s
2017-05-07 01:55:51 (209 KB/s) - ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’ saved [65536/65536]
--2017-05-07 01:55:51-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless;type=a
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/plain]
Saving to: ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’
COCO.IS.EpiSensor.100hz.860384.da [ <=> ] 64.00K 208KB/s in 0.3s
2017-05-07 01:55:54 (208 KB/s) - ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’ saved [65536]
--2017-05-07 01:55:54-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless;type=i
Proxy request sent, awaiting response... 200 Gatewaying
The file is already fully retrieved; nothing to do.
--2017-05-07 01:55:56-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless
Proxy request sent, awaiting response... 200 Gatewaying
Length: 32768 (32K) [text/plain]
Saving to: ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’
COCO.IS.borehole.20hz.278674.data 100%[============================================================>] 32.00K 210KB/s in 0.2s
2017-05-07 01:55:58 (210 KB/s) - ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’ saved [32768/32768]
--2017-05-07 01:55:58-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless;type=a
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/plain]
Saving to: ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’
COCO.IS.borehole.20hz.278674.data [ <=> ] 32.00K 208KB/s in 0.2s
2017-05-07 01:56:01 (208 KB/s) - ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’ saved [32768]
--2017-05-07 01:56:01-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless;type=i
Proxy request sent, awaiting response... 200 Gatewaying
The file is already fully retrieved; nothing to do.
FINISHED --2017-05-07 01:56:03--
Total wall clock time: 17s
Downloaded: 5 files, 197K in 0.9s (215 KB/s)

Related Links

How to add text at the end of each line in unix
sed whole word search and replace but include other characters
Using shell script to copy script from one file to another
Unix Shell Script to take multiple files from standard input (csh)
Calculation using expr in shell
Unset IFS value for shell script
Get all running ports using UNIX command and form a new command based on result
Running R script from Shell using CygWin: error “Rscript not found”
How to get the return code of a shell script in lua?
Get Key Names from jq JSON UNIX command
selecting different values from two lines in shell
Shell script: changing text file without knowing the original value
How to save shell screen to file verbose?
IFS=$'\n' doesn't change IFS to breaklines
GetOpt-like Behavior in PowerShell
How to run a .cmd file through Jenkins?

Categories

HOME
httparty
ns-3
jsf-2.2
devise
spring-data-redis
appcelerator
switch-statement
emscripten
pyqt4
vsm
modal-dialog
performancepoint
pcl-crypto
dkim
resultset
radgridview
code-generation
enterprise-architect
libusb
altera
xmonad
shortest-path
pchart
event-store
tosca
interpreter
draft-js-plugins
moses
interbase
fragment-backstack
inversion-of-control
android-things
ceylon
gp
gitlab-api
recurring-events
sql-server-2000
code-snippets
linkedhashmap
servicestack-text
swagger-php
watchman
google-earth-engine
apns-php
c9.io
vmware-fusion
plr
avcapturedevice
contract
phpdbg
pom.xml
magento-2.0.7
tinymce-3
openblas
trojan
video-conferencing
git-flow
mach-o
mathcad
jtree
grails-plugin
centrifuge
jtwig
spreadjs
halcon
pushbots
aspen
java-money
file-move
pyobjc
mraid
osc
metatrader5
iso-prolog
webfinger
appserver
insight.database
grails-plugin-rabbitmq
mutual-authentication
maven-scm
light
embedded-fonts
breakout
scringo
chart-director
tree-balancing
uimanageddocument
appjs
jdb
promotion-code
mysql-error-1205
automationelement
xui
http-daemon
load-time
instantiationexception
cruisecontrol.rb
silent
aspmenu
web-garden
lxr
mathematical-notation
ifilter
information-management
idatareader

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App