I don’t need to stress how important documentation is to ensure repeatability of work in any field; even the best of us will forget what we did last week and waste time recreating the same (hopefully!) solution. Luckily in IT, a shell script can perform those repetitive tasks AND document the solution at the same time…provided you write readable code and/or comment well!
As SmartFile’s Linux Administrator, when I’m not helping diagnose customer issues, I usually find myself doing a lot of repetitive tasks. Occasionally, a customer will have a question that really peaks my scripting interests. This is one such case.
A customer asked:
“I’m attempting to write a script that retrieves files from specific directories, but using NcFTPGet and MGET have both returned “No such file or directory” when I am certain that the files I’m attempting to retrieve exist in these directories. Below is the command attempted with NcFTPGet:ncftpget -DD -u USERNAME -p PASSWORD app.smartfile.com ~/downloads /customer/2111/*.csv Any idea why NcFTPGet or MGET would fail? Thanks.”
How I Found the Answer
Well, I wasn’t familiar at all with NcFTPGet command (there are a lot of FTP programs out there!) but this seems like something that should work, doesn’t it? I quickly discovered that the NcFTP client is available via Yum on my Fedora 18 machine (and most other platforms through the NcFTP website).
Checking out the main page, I determined that the command should download any .csv file in the /customer/2111/ folder, and then remove them from the server. Unfortunately, the NcFTP program (and the BSD FTP binary) MGET (multiple file get) command relies on server side wildcard matching, which is not part of the FTP specification, and our server returns the error that it can’t find a file named “*.csv”. Some FTP services DO support wildcards, but the implementation is not uniform, so I would consider it best practice to use client-side wildcard matching.
Enter the Kermit Project, a 30-plus-year-old set of file transfer and communication protocols that was once the de facto standard in communications software. Kermit’s included FTP client supports client-side wildcard matching and has many more features that you may not have known you were missing in other FTP clients. Kermit is available as the ‘ckermit’ package in yum on Fedora, and is readily available for many other systems as well.
How I Applied What I Learned
Let’s just start with a quick example in Kermit that should look very similar to scripting other FTP clients, which downloads some images to my local machine and removes them from the server:
#!/usr/bin/kermit +cd /home/akubacki/downloadsftp open app.smartfile.com /user:akubacki /password:XXXXXXXXcd uploadedimagesmget /delete /mlsd *.gifexit
The /delete option will remove the file from the server once downloaded, the ‘/mlsd‘ option forces Kermit to get a list of ALL the files in the folder, and then apply the wildcard locally to only get/delete the files we want to. Bingo! At this point, it’s worth taking the time to develop a general solution (let’s call that script mgetdelete.kermit) that mimics the NcFTP command from above.
Usage : ./mgetdelete.kermit ftp.server.url username password remote-folder file(s) local-folder
#!/usr/bin/kermit +cd \%6if fail exit 1 Local destination errorftp open \%1 /user:\%2 /password:\%3if fail exit 1 Connection failedcd \%4if fail exit 1 Source directory error: \v(ftp_message)mget /delete /mlsd \%5if fail exit 1 mget/delete error: \v(ftp_message)ftp byeexit
As you can see, I’ve added some error handling to make sure you don’t download the wrong files, to the wrong location, and then delete them from the server! Kermit has the scripting ability more similar to bash/csh shells than traditional FTP clients, so the possibilities are near endless to add logic to an automated FTP session! There are some other great examples on the Kermit project website to get you started!
And don’t forget what I said about commenting scripts thoroughly so it can double as documentation, I certainly should have done better in my example!
Image Credit: Bruno Oliveira, XKCD.com