e-Zest members share technology ideas to foster digital transformation.

Downloading Large Files through Web Service

Written by Nikhil Vakil | Sep 6, 2012 4:12:47 PM

This solution is developed in Java. The solution can be implemented where any File Download tool is not to be used.

Requirement:

Consider two applications:

  1. Web Application (WA): This application includes File Download Web Service (FDWS) for File Download
  2. Desktop Application (DA): This application is installed at multiple locations.

    Download large file from WA to DA. The files are different per user.

    If file download fails then next time

    • Only the remaining file should be downloaded.
    • But not the complete file.
Note:
  1. Max file size supported = Java int range.
  2. Files upto 19 MB are tested with this solution.
  3. The solution is still under testing and is subject to change.
Design Consideration:

The WA application breaks Large File in five files. The DA invokes file threads to write these files.
Each thread

  1. Reads one of the file files from WA
  2. Writes the content on desktop.

Once, all threads finish downloading then all files are combined in one file.

Pre-requisite:
  1. WA breaks large file into five different files.
    • Large File Size = 100 MB
    • Name of the file “Data.Zip”
    • Create five files of equal size. Each file size = 20 MB.
    • Name files in a way to define the order. For example.
      1. Data.Zip_0
      2. Data.Zip_1
      3. Data.Zip_2
      4. Data.Zip_3
      5. Data.Zip_4
  2. WA intimates DA that data is read and DA can start downloading. This step may vary in different applications.
Implementation Guidelines:
  1. DA opens threads to read different files. See Figure 1
    • Create an object that extends Thread.
    • Five threads are opened to read five files. Refer to Pre-requisite section 1.d.
    • Each thread
      • Is instantiated with an Index number starting from 0 to 4.
      • Creates a file on desktop with same Index.
      • Creates a FileOutputStream object that points to this file.
    • Each thread calls FDWS.
    • Thread sends Index number and User Id with the request.
  2. DA creates Timer and Timer Task to watch threads. See Figure 1
    • This timer is scheduled to execute after every 5 seconds.
    • The timer keeps checking if all threads have finished reading or not.
    • If thread is finished reading then
      • Timer is cancelled.
      • WA is informed that downloading is complete .
      • All files are combined to create one file.
  3. FDWS receives request from DA. See Figure 1
    • Receives the request from CA.
    • Reads Index and User Id from requests.
    • Creates a File Object that matches the Index.
    • Creates a FileInputStream object with this File.
    • This FileInputStream object is maintained in a class variable.
    • The variable is Map>
    • The FDWS will be receiving requests from multiple threads. Hence, User Id should be synchronized.
      Updating above Map should be done in this synchronized block.
  4. FDWS reads the file and sends data to DA. See Figure 1
    • Reads 50000 bytes from FileInputStream.
    • It uses int FileInputStream.read(byte[] bytes) method.
    • The read bytes are sent to DA.
    • It also intimates status File Reading Not Finished.
    • However, if the FileInputStream object finished reading file, then it intimates status File Reading Finished.
  5. A thread in DA writes data in its own file. See Figure 2
    • If thread receives File Reading Not Finished then it writes data to a File with FileOutputStream object.
    • If thread receives File Reading Finished then
      • The thread is interrupted.
      • Updates flag that file reading is finished.
      • Timer instantiated above checks for the same flag.
  6. Once finished, DA combines all files and creates one file. See Figure 3
  7. In case if any thread receives an error, then all threads are aborted.
  8. DA calls another Web Service that informs WA that download is stopped.
  9. Next time when threads start, it carries the amount of bytes downloaded.
  10. WA skips these numbers of bytes from file and starts reading the file.
  11. WA keeps checking for request from User Ids. If request from a user is not received for 5 minutes then all FileInputStream objects are removed from memory.

Figure 1

Figure 2

Figure 3
View full post