shell


wget downloads files from FTP site TWICE


I'm downloading files under a folder on a FTP site using
wget -A gz -rc -np -nd -l1 -t0 ftp://path/to/folder/
wget downloads index.html first (and then parses urls from it?) and then rejects it.
I checked the links in index.html. For each file, there are 3 links to it:
href=file.gz
href=file.gz;type=a
href=file.gz;type=i
The file at ftp://path/to/folder/file.gz is downloaded and saved to file.gz. Then the file at ftp://path/to/folder/file.gz;type=a is also downloaded and saved to the same file (a redundant download).
Since type=i in defaults, the file at ftp://path/to/folder/file.gz;type=i is the same as the first one and no file transferring actually happens.
How can I avoid the downloading of file.gz;type=a? I tried to add option -R "type=a" but it doesn't work.
Here is an example:
$ wget -A "C*.dataless" -rc -np -nd -l1 -P "/tmp/" ftp://ds.iris.edu/pub/userdata/Ed_Anderson/
--2017-05-07 01:55:46-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/
Proxy request sent, awaiting response... 200 OK
Length: 5522 (5.4K) [text/html]
Saving to: ‘/tmp/index.html’
index.html 100%[============================================================>] 5.39K --.-KB/s in 0s
2017-05-07 01:55:49 (178 MB/s) - ‘/tmp/index.html’ saved [5522/5522]
Removing /tmp/index.html since it should be rejected.
--2017-05-07 01:55:49-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless
Proxy request sent, awaiting response... 200 Gatewaying
Length: 65536 (64K) [text/plain]
Saving to: ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’
COCO.IS.EpiSensor.100hz.860384.da 100%[============================================================>] 64.00K 209KB/s in 0.3s
2017-05-07 01:55:51 (209 KB/s) - ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’ saved [65536/65536]
--2017-05-07 01:55:51-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless;type=a
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/plain]
Saving to: ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’
COCO.IS.EpiSensor.100hz.860384.da [ <=> ] 64.00K 208KB/s in 0.3s
2017-05-07 01:55:54 (208 KB/s) - ‘/tmp/COCO.IS.EpiSensor.100hz.860384.dataless’ saved [65536]
--2017-05-07 01:55:54-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.EpiSensor.100hz.860384.dataless;type=i
Proxy request sent, awaiting response... 200 Gatewaying
The file is already fully retrieved; nothing to do.
--2017-05-07 01:55:56-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless
Proxy request sent, awaiting response... 200 Gatewaying
Length: 32768 (32K) [text/plain]
Saving to: ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’
COCO.IS.borehole.20hz.278674.data 100%[============================================================>] 32.00K 210KB/s in 0.2s
2017-05-07 01:55:58 (210 KB/s) - ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’ saved [32768/32768]
--2017-05-07 01:55:58-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless;type=a
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/plain]
Saving to: ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’
COCO.IS.borehole.20hz.278674.data [ <=> ] 32.00K 208KB/s in 0.2s
2017-05-07 01:56:01 (208 KB/s) - ‘/tmp/COCO.IS.borehole.20hz.278674.dataless’ saved [32768]
--2017-05-07 01:56:01-- ftp://ds.iris.edu/pub/userdata/Ed_Anderson/COCO.IS.borehole.20hz.278674.dataless;type=i
Proxy request sent, awaiting response... 200 Gatewaying
The file is already fully retrieved; nothing to do.
FINISHED --2017-05-07 01:56:03--
Total wall clock time: 17s
Downloaded: 5 files, 197K in 0.9s (215 KB/s)

Related Links

check for newline character in a csv file
U1077 Error running nmake on live555
compare two files unix and print data
Iterating over characters in a string in POSIX sh
Different behavior on escaping double quotes
How to merge two files with a common key in awk?
shell date “-n hours” differs with “n hours ago” in some situation
How to apply loop in two files in awk with one matched column?
How to merge log files and sort by time
What's the difference between cat and ypcat?
Check the current shell and update user profiles using shell script
How to develop this sort script further
How to Find & Replace Strings with Incrementing Variables in a Text File Using Awk
Undefined property in cakePHP shell
How do i multiply decimal point in shell
Bash script loop break even with true statement

Categories

HOME
service-worker
deep-learning
linkedin
phpmyadmin
aws-cli
rds
mapbox
retrofit2
crystal-lang
plist
naivebayes
qsqlquery
aws-sdk
foreign-keys
symfony-console
database-connection
guzzle
named-entity-recognition
r-mice
ios10.2
resourcebundle
onload
delimiter
mat
ms-access-2003
tpm
record
database-comparison
wacom
clickjacking
dynamic-jasper
slacktextviewcontroller
imagenet
jpad
railstutorial.org
dcmtk
branching-and-merging
preloader
android-exoplayer
jide
sonarlint-eclipse
dtsearch
tf
datamaps
pair-programming
amq
video-conferencing
mach-o
monkey
jtree
choice
adafruit
synonym
data-protection
uiactionsheet
pushbots
boomerang
themoviedb-api
toolkit
user-permissions
dot.js
xmltype
public-html
pseudo-class
ss7
alternate
tinyxml
authlogic
websphere-esb
light
model-driven-development
diem
swfobject
redis-py
vfs
mapping-by-code
azman
failing-tests
cgimage
flex-mobile
user-tracking
winmain
ie-compatibility-mode
web-garden
filesystemobject
fdf
netbeans6.1

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App