shell


wget downloads files from FTP site TWICE


I'm downloading files under a folder on a FTP site using
wget -A gz -rc -np -nd -l1 -t0 ftp://path/to/folder/
wget downloads index.html first (and then parses urls from it?) and then rejects it.
I checked the links in index.html. For each file, there are 3 links to it:
href=file.gz
href=file.gz;type=a
href=file.gz;type=i
The file at ftp://path/to/folder/file.gz is downloaded and saved to file.gz. Then the file at ftp://path/to/folder/file.gz;type=a is also downloaded and saved to the same file (a redundant download).
Since type=i in defaults, the file at ftp://path/to/folder/file.gz;type=i is the same as the first one and no file transferring actually happens.
How can I avoid the downloading of file.gz;type=a? I tried to add option -R "type=a" but it doesn't work.
Here is an example:
$ wget -A "C*.dataless" -rc -np -nd -l1 -P "/tmp/" ftp://ds.iris.edu/pub/userdata/Ed_Anderson/
--2017-05-07 01:55:46-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/
Proxy request sent, awaiting response... 200 OK
Length: 5522 (5.4K) [text/html]
Saving to: ‘/tmp/index.html’
index.html 100%[============================================================>] 5.39K --.-KB/s in 0s
2017-05-07 01:55:49 (178 MB/s) - ‘/tmp/index.html’ saved [5522/5522]
Removing /tmp/index.html since it should be rejected.
--2017-05-07 01:55:49-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless
Proxy request sent, awaiting response... 200 Gatewaying
Length: 65536 (64K) [text/plain]
Saving to: ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’
COCO.IS.EpiSensor.100hz.860384.da 100%[============================================================>] 64.00K 209KB/s in 0.3s
2017-05-07 01:55:51 (209 KB/s) - ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’ saved [65536/65536]
--2017-05-07 01:55:51-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless;type=a
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/plain]
Saving to: ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’
COCO.IS.EpiSensor.100hz.860384.da [ <=> ] 64.00K 208KB/s in 0.3s
2017-05-07 01:55:54 (208 KB/s) - ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’ saved [65536]
--2017-05-07 01:55:54-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless;type=i
Proxy request sent, awaiting response... 200 Gatewaying
The file is already fully retrieved; nothing to do.
--2017-05-07 01:55:56-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless
Proxy request sent, awaiting response... 200 Gatewaying
Length: 32768 (32K) [text/plain]
Saving to: ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’
COCO.IS.borehole.20hz.278674.data 100%[============================================================>] 32.00K 210KB/s in 0.2s
2017-05-07 01:55:58 (210 KB/s) - ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’ saved [32768/32768]
--2017-05-07 01:55:58-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless;type=a
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/plain]
Saving to: ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’
COCO.IS.borehole.20hz.278674.data [ <=> ] 32.00K 208KB/s in 0.2s
2017-05-07 01:56:01 (208 KB/s) - ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’ saved [32768]
--2017-05-07 01:56:01-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless;type=i
Proxy request sent, awaiting response... 200 Gatewaying
The file is already fully retrieved; nothing to do.
FINISHED --2017-05-07 01:56:03--
Total wall clock time: 17s
Downloaded: 5 files, 197K in 0.9s (215 KB/s)

Related Links

how to pipe commands in ubuntu
Add directory to system path in OS X Lion
GDB: Do not print output of .gdbinit commands
Problems with shell scriptings using Sed
Why uniq -c output with space instead of \t?
“ls” command output [closed]
No command executed after performing kill command in shell script
replacing the string of the file name and using it in the same shell script
Unix for loop skipping one value in script, works properly in command line
HBase Shell hangs / freezes
In memory storage of blocks in shell script
Unix: removing duplicate lines without sorting
Using scala.sys.process to invoke echo on Windows 7
shell tab-completion of strings based on more than just the first characters?
Is there a portable way using unix shell commands to tell if there is a configured IPv6 interface?
Execute shell script from spring web application deployed on cloudfoundry

Categories

HOME
sonarqube
django
asp.net-web-api
google-app-engine
paraview
time-complexity
serialization
graphics
gradient
sublimetext
newrelic
x264
jquery-ui
python-multithreading
retrofit2
bing-maps
mql5
github-enterprise
plist
atlassian
magnetic-cards
alpacajs
production
sencha-touch
lilypond
simpleitk
r-mice
joptionpane
game-theory
delimiter
underscore.js
angular-translate
complex-numbers
mustache
left-join
cart
glib
memory-address
biztalk-rule-engine
protege4
graphhopper
odroid
google-data-api
mongoose-schema
spring-mvc-test
kill
sharp-snmp
subclipse
dynamic-linq
podscms
payara-micro
py2neo
jtree
reachability
donations
smartfoxserver
pure-css
angular2-testing
latin
signalr.client
i386
spreadjs
highland.js
tfs-workitem
zen-cart
android-launcher
eoferror
skspritenode
airbrake
aspectj-maven-plugin
mraid
j-security-check
shopizer
beatsmusic
imagefilter
cbind
jama
mobility
memset
buffering
openargs
artemis
c++builder-xe5
windows-phone-8-sdk
cnf
poker
time.h
scmmanager
onload-event
insert-id
android-sdk-2.3
device-width
vfs
xcode3.2
os.system
subgraph
jquery-address
ikimagebrowserview
saxparseexception
diagrams
aspmenu
addchild
nstokenfield
wordprocessingml
fault-tolerance
product-management
change-management
online-storage

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App