Transparent background file download from S3

I am trying to permit a python app to access various locations in a many-GB file stored in S3. I'd like to create a drop-in replacement file-like object that intelligently downloads chunks of data from S3 in a separate thread to meet seek() and read() requests.

Is there a simple data structure I can use to store arbitrary intervals of the file?

It must support O(log n) look-up and O(n) insertion (n=number of chunks, not size of file). It will also need to support quickly querying for gaps so that the load thread can efficiently find the next chunk it should download. This is currently not supported by things like SortedCollection, suggesting I may need to manually use bisect_* in a new container.

Example usage is:

import os
import time
from bigfile import BigFile

chunksize = (2**20)*64 # 64MB

bf = BigFile('my_bucket', 'key_name', chunksize=chunksize)

# read from beginning (blocks until first chunk arrives)
bf.read(100)

# continues downloading subsequent chunks in background
time.sleep(10)

# seek into second chunk and read (should not block)
bf.seek(blocksize, os.SEEK_SET)
bf.read(100)

# seek far into the file
bf.seek(blocksize*100 + 54, os.SEEK_SET) # triggers chunk download starting at new location
bf.read(100) # blocks until chunk arrives

# seek back to beginning (should not block, already have this chunk)
bf.seek(0, os.SEEK_SET)
bf.read(100)

# read entire rest of file (blocks until all chunks are downloaded)
bf.read()


Asked by: Kelvin711 | Posted: 30-11-2021






Answer 1

This implementation uses chunks of fixed size and offsets. If the chunks are very large and the network is very slow, reads may block for a long time (consider a read starting at the last byte of a chunk, it would have to wait for the entire previous chunk to load, then the next chunk).

Ideally we could use chunks of arbitrary size and location, so we can optimize loads to start at exactly the read point. But below is a good 80% solution.

import boto
import threading
import tempfile
import os

DEFAULT_CHUNK_SIZE = 2**20 * 64 # 64 MB per request

class BigFile(object):
    def __init__(self, file_obj, file_size, chunksize=DEFAULT_CHUNK_SIZE, start=True):
        self._file_obj = file_obj
        self._file_size = file_size
        self._lock = threading.RLock()
        self._load_condition = threading.Condition(self._lock)
        self._load_run = True
        self._loc = 0
        self._chunk_size = chunksize
        chunk_count = self._file_size // self._chunk_size
        chunk_count += 1 if self._file_size % self._chunk_size else 0
        self._chunks = [None for _ in xrange(chunk_count)]
        self._load_thread = threading.Thread(target=self._load)
        if start:
            self._load_thread.start()

    def _chunk_loc(self):
        ' Returns (chunk_num, chunk_offset) for a given location in the larger file '
        return self._loc // self._chunk_size, self._loc % self._chunk_size

    def _load_chunk(self, chunk_num):
        tf = tempfile.TemporaryFile()
        start_idx = chunk_num * self._chunk_size
        self._file_obj.seek(start_idx)
        tf.write(self._file_obj.read(self._chunk_size))
        with self._lock:
            self._chunks[chunk_num] = (tf, tf.tell()) # (tempfile, size)
            self._load_condition.notify()

    def _load(self):
        while self._load_run:
            # check current chunk, load if needed
            with self._lock:
                chunk_num, _ = self._chunk_loc()
                chunk_and_size = self._chunks[chunk_num]
            if chunk_and_size is None:
                self._load_chunk(chunk_num)

            # find next empty chunk
            for i in xrange(len(self._chunks)):
                cur_chunk = chunk_num + i
                    cur_chunk %= len(self._chunks) # loop around
                if self._chunks[cur_chunk] is None:
                    self._load_chunk(cur_chunk)
                    break
            else:
                # all done, stop thread
                break

    def seek(self, loc, rel=os.SEEK_SET):
        with self._lock:
            if rel == os.SEEK_CUR:
                self._loc += loc
            elif rel == os.SEEK_SET:
                self._loc = loc
            elif rel == os.SEEK_END:
                self._loc = self._file_size + loc

    def read(self, bytes_to_read):
        ret = []
        with self._lock:
            chunk_num, chunk_offset = self._chunk_loc()
            while (bytes_to_read > 0 or bytes_to_read == -1) and chunk_num < len(self._chunks):
                while not self._chunks[chunk_num]:
                    self._load_condition.wait()
                chunk, size = self._chunks[chunk_num]
                cur_chunk_bytes = min(self._chunk_size-chunk_offset, bytes_to_read, size)
                chunk.seek(chunk_offset, os.SEEK_SET)
                data = chunk.read(cur_chunk_bytes)
                ret.append(data)
                bytes_to_read -= len(data)
                chunk_num += 1
        return ''.join(ret)

    def start(self):
        self._load_thread.start()

    def join(self):
        self._load_thread.join()

    def stop(self):
        self._load_run = False

class S3RangeReader:
    def __init__(self, key_obj):
        self._key_obj = key_obj
        self.size = self._key_obj.size
        self._pos = 0

    def __len__(self):
        return self.size

    def seek(self, pos, rel=os.SEEK_SET):
        if rel == os.SEEK_CUR:
            self._pos += pos
        elif rel == os.SEEK_SET:
            self._pos = pos
        elif rel == os.SEEK_END:
            self._pos = self.size + pos

    def read(self, bytes=-1):
        if bytes == 0 or self._pos >= self.size:
            return ''
        else:
            if bytes == -1:
                bytes = self.size
            headers = {'Range': 'bytes=%s-%s' % (self._pos, self._pos + bytes - 1)} # S3 ranges are closed ranges: [start,end]
            return self._key_obj.get_contents_as_string(headers=headers)

if __name__ == '__main__':
    key = boto.s3_connect().get_bucket('mybucket').get_key('my_key')
    reader = S3RangeReader(key)
    bf = BigFile(reader, len(reader)) # download starts by default
    bf.seek(1000000)
    bf.read(100) # blocks
    bf.seek(0)
    bf.read(100) # should not block

Answered by: Dainton965 | Posted: 01-01-2022



Similar questions

python - How to make a surface with a transparent background in pygame

Can someone give me some example code that creates a surface with a transparent background in pygame?


django - python PIL - background displayed opaque instead of transparent

I want to generate 32x32 sized thumbnails from uploaded images (actually avatars). To prevent a thumbnail from being smaller than that size, I want to create a transparent 32x32 background and paste the thumbnail on it. The code below tries to do so. However, the avatar is displayed on a black and opaque background; I lose transparency information somewhere through the process. Where am I doing wrong?...


How do I create a Status Icon / System Tray Icon with custom text and transparent background using Python and GTK?

Here is the code that I have so far to define the icon: icon_bg = gtk.gdk.pixbuf_new_from_file('gmail.png') w, h = icon_bg.get_width(), icon_bg.get_height() cmap = gtk.gdk.Colormap(gtk.gdk.visual_get_system(), False) drawable = gtk.gdk.Pixmap(None, w, h, 24) drawable.set_colormap = cmap gc = drawable.new_gc() drawable.draw_pixbuf(gc, icon_bg, 0, 0, 0, 0, w, h) drawn_icon = gtk.gdk.Pixbuf(gtk.gdk.COLORSPAC...


python - How to get transparent background in window with PyGTK and PyCairo?

I've been trying really hard to create a window with no decoration and a transparent background using PyGTK. I would then draw the content of the window with Cairo. But I can't get it to work. I've tried a lot of different ways, they all failed, this is one of them #!/usr/bin/env python import pygtk pygtk.require('2.0') import gtk, sys, cairo win = None def expose (widget, event): cr = widget...


White background to transparent background using PIL python

How can i transform all white background and white elements of a png or jpg image in a transparent backgroun using PIL?


java - Detect if .swf has transparent background

Is there a way to do this programmatically in PHP, Python or Java? Use case: User uploads .swf through an upload form. Detect if it has a transparent background. If it does, change it to something else, e.g. white.


user interface - I want to make a transparent BACKGROUND frame with my python app (WxPython)

I've always wanted to make a app that's transparent, I love how it looks. I tried to make on of my apps transparent with the SetTransparent(alpha), but I bumped into a big problem. Yes it makes my app transparent (^。^), but it makes EVERYTHING transparent (=。=). I have no idea how to fix this problem. Hours of research didn't really help out. What I meant by background is, nothing el...


c++ - How to make wxPython, DC background transparent?

How can i make the default white DC background become invisible (transparent)? Run the following example, it shows a white background over a button. I would like to remove the white background of the DC. (to show only the red X) import wx class drawover(wx.Window): def __init__(self, parent): wx.Window.__init__(self, parent) self.Bind(wx.EVT_PAINT, self.OnPaint) self....


python - Merging background with transparent image in PIL

I have a png image as background and I want to add a transparent mesh to this background but this doesn't work as expected. The background image is converted to transparent on places where I apply transparent mesh. I am doing: from PIL import Image, ImageDraw map_background = Image.open(MAP_BACKGROUND_FILE).convert('RGBA') map_mesh = Image.new('RGBA', (width, height), (0, 0, 0, 0)) draw = ImageDraw...


python - Transparent background in gnuplot multiplot

I'd like to get a transparent background in my gnuplot figure. What I coded till now is: #! /usr/bin/python from numpy import * import Gnuplot as gp import Gnuplot.funcutils x = (1,2,3) y=(2,4,5) x1 = (3,6,8) g = gp.Gnuplot() g("set terminal svg size 200,400") g("set output 'filename.svg'") g("unset xtics") g("unset ytics") g("set multiplot layout 3,1 title 'Title here'") g("set object 1 rectangle from ...


python - How to make a surface with a transparent background in pygame

Can someone give me some example code that creates a surface with a transparent background in pygame?


python - How to use PIL to make all white pixels transparent?

I'm trying to make all white pixels transparent using the Python Image Library. (I'm a C hacker trying to learn python so be gentle) I've got the conversion working (at least the pixel values look correct) but I can't figure out how to convert the list into a buffer to re-create the image. Here's the code img = Image.open('img.png') imga = img.convert("RGBA") datas = imga.getdata() newData = list() for i...


pygame - Updating part of a surface in python, or transparent surfaces

I have an application written in python that's basically an etch-a-sketch, you move pixels around with WASD and arrow keys and it leaves a trail. However, I want to add a counter for the amount of pixels on the screen. How do I have the counter update without updating the entire surface and pwning the pixel drawings? Alternatively, can I make a surface that's completely transparent except for the text so you can se...


django - python PIL - background displayed opaque instead of transparent

I want to generate 32x32 sized thumbnails from uploaded images (actually avatars). To prevent a thumbnail from being smaller than that size, I want to create a transparent 32x32 background and paste the thumbnail on it. The code below tries to do so. However, the avatar is displayed on a black and opaque background; I lose transparency information somewhere through the process. Where am I doing wrong?...


python - Transparent FrameBuffer background in OpenGL

I want to use glClear and glClearColor to fill a frame buffer with a colour including alpha transparency. However the framebuffer always renders as opaque when binded to a texture which is rendered to the screen. I want everything which is rendered to the framebuffer to kept their transparency. I just want to change the background. See the following code: def create_texture(surface): surface...


How do I create a Status Icon / System Tray Icon with custom text and transparent background using Python and GTK?

Here is the code that I have so far to define the icon: icon_bg = gtk.gdk.pixbuf_new_from_file('gmail.png') w, h = icon_bg.get_width(), icon_bg.get_height() cmap = gtk.gdk.Colormap(gtk.gdk.visual_get_system(), False) drawable = gtk.gdk.Pixmap(None, w, h, 24) drawable.set_colormap = cmap gc = drawable.new_gc() drawable.draw_pixbuf(gc, icon_bg, 0, 0, 0, 0, w, h) drawn_icon = gtk.gdk.Pixbuf(gtk.gdk.COLORSPAC...


python - How to make cStringIO transparent to another function that expects a real local file

I came up with the following problem: CODE A works right now.. I am saving a png file called chart.png locally, and then I am loading it into the proprietary function (which I do not have access). However, in CODE B, am trying to use cStringIO.StringIO() so that I do not have to write the file "chart.png" to the disk. But I cannot find a way to pass it to the pproprietaryfunction because it is expecting a real fil...


python - PIL: How to make area transparent in PNG?

I've been using PIL to crop Images, now I also want to make certain rectangular areas transparent, say from PIL import Image im = Image.open(&quot;sample.png&quot;) transparent_area = (50,80,100,200) ...


python - Matplotlib transparent line plots

I am plotting two similar trajectories in matplotlib and I'd like to plot each of the lines with partial transparency so that the red (plotted second) doesn't obscure the blue. EDIT: Here's the image with transparent lines.


python - Transparent 3D bar graphs

I would like to generate 3D bar graphs with transparent surfaces so that I can see what is going on behind tall bars. The mplot3d API docs say that keywords are allowed for the bar3d function. I pass all the required parameters but can only output graphs with solid surfaces.






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top