Using FTP to Sync

I've needed a script to sync a remote folder, its a folder that is used to get updates dropped into it but I wanted to download only non existing files, I've found a script on dostips, which I've used for a while until I've stumbled on their note "Since all files are passed into the FTP`s MGET command there might be a limit to the number of files that can be processed at once."

So I've changed the script, it batches downloads, you can specify how many files to download in a batch, but 10 is the most stable (it depends on the filename length).

@Echo Off
REM Taken from http://www.dostips.com/DtTipsFtpBatchScript.php
REM 2011-02-15 - Dror Gluska - Added limit of files to download in a batch

if [%1]==[] goto usage

set servername=%~1
set username=%~2
set password=%~3
set rdir=%~4
set ldir=%~5
set filematch=%~6
set maxfilesperbatch=%~7


REM -- Extract Ftp Script to create List of Files
Set "FtpCommand=ls"
Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"
rem notepad "%temp%\%~n0.ftp"

REM -- Execute Ftp Script, collect File Names and execute batch download
setlocal ENABLEDELAYEDEXPANSION

set /a nocount=0
set /a totalfiles=0

Set FileList=
For /F "tokens=* delims= " %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %filematch%"') Do (
    call set filename=%%~A
    
    if Not Exist "%ldir%\!filename!" (
        echo [!filename!] added to batch
    
        set /a nocount+=1
        set /a totalfiles+=1
        call Set FileList=!FileList! ""!filename!""
        
        if !nocount! EQU !maxfilesperbatch! (
            
            if !nocount! gtr 0 (
                echo Downloading !totalfiles! files...
                Call:downloadFiles "!FileList!"
                call set /a nocount=0
                call Set FileList=
            )
        )
    )
)

if !nocount! gtr 0 (
    echo Downloading !totalfiles! files...
    Call:downloadFiles "%FileList%"
)

endlocal

exit /B 0

GOTO:EOF

:downloadFiles filenames
SETLOCAL Disabledelayedexpansion
Set "FtpCommand=mget "
call set "FtpCommand=%FtpCommand% %~1"

call set FtpCommand=%FtpCommand:""="% 

rem echo %nocount% files to download

Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"
   rem notepad "%temp%\%~n0.ftp"

REM -- Execute Ftp Script, download files

ftp -i -s:"%temp%\%~n0.ftp" > nul
Del "%temp%\%~n0.ftp"

exit /b

:usage

echo %0 ^<server^> ^<username^> ^<password^> ^<remotepath^> ^<localpath^> ^<maximum files^>
exit /B 1

goto:EOF


:extractFileSection StartMark EndMark FileName -- extract a section of file that is defined by a start and end mark
::                  -- [IN]     StartMark - start mark, use '...:S' mark to allow variable substitution
::                  -- [IN,OPT] EndMark   - optional end mark, default is first empty line
::                  -- [IN,OPT] FileName  - optional source file, default is THIS file
:$created 20080219 :$changed 20100205 :$categories ReadFile
:$source http://www.dostips.com
SETLOCAL Disabledelayedexpansion
set "bmk=%~1"
set "emk=%~2"
set "src=%~3"
set "bExtr="
set "bSubs="
if "%src%"=="" set src=%~f0&        rem if no source file then assume THIS file
for /f "tokens=1,* delims=]" %%A in ('find /n /v "" "%src%"') do (
    if /i "%%B"=="%emk%" set "bExtr="&set "bSubs="
    if defined bExtr if defined bSubs (call echo.%%B) ELSE (echo.%%B)
    if /i "%%B"=="%bmk%"   set "bExtr=Y"
    if /i "%%B"=="%bmk%:S" set "bExtr=Y"&set "bSubs=Y"
)
EXIT /b


[Ftp Script 1]:S
!Title Connecting...
open %servername%
%username%
%password%

!Title Preparing...
cd %rdir%
lcd %ldir%
binary
hash

!Title Processing... %FtpCommand%
%FtpCommand%

!Title Disconnecting...
disconnect
bye

Example:

ftpsync 10.0.0.1 "anonymous" "email@email.com" "/" ".\Temp" ".txt" 10

You have to specify all the parameters.

Unfortunately you have to specify some kind of regex file match, otherwise it will attempt to download the ftp status messages too, could be a problem if there are more status messages than number of files in a batch.


Tags: