Backpack V0.0.4 - An Entire Rewrite

Table of Contents

Backpack v0.0.4 - An Entire Rewrite

I started learning how to code in Python. Lately I’ve been coding in Go and it’s not only been fun but I’ve learned a lot. There are some practices learned in Go that I’ve been waiting to try out in some Python code. Accordingly I decided to take another look at my primitive backup program backpack. The changes I’ve made thus far are early stages but I like the way things are looking thus far.

v0.0.4-1 Changes

Dataclasses

A tutorial article passed by my Google News feed which was an introduction into Python’s dataclass objects. A dataclass is like a class object without any function - it is simply initialized to store data. For example in Backpack I have two dataclasses: File() and Directory().

@dataclass(frozen=True)
class File:
    name: str
    path: str       # full path
    size: int       # add repr to do kb conversion - round(stat.st_size/(1024**2),3)
    owner: str      # uid/gid
    hash: str       # md5

@dataclass(unsafe_hash=True)
class Directory:
    name: str
    path: str
    subDirs: list
    files: list[File]

    def __post_init__(self):
        dir_objs = []
        for d in self.subDirs:
            dir_objs.append(scan_dir(d))
        self.__setattr__("subDirs", dir_objs)

File() Object

You’ll notice in the file definition the @dataclass decorator includes the flag frozen=True. This indicates that post initialization of the object that it should be immutable.

Most of the File attributes are supplied by the DirEntry object which is the return object when using os.scandir(fp). The hash attribute is filled via a sub-function of scan_dir(). After determining the item/path is a file it calls the generate hash function which issues a subprocess call to md5sum.

💡 IDEA - move the generate hash function from scan_dir() to in a __post_init__() call within File()

Directory() Object

The directory object is pretty neat imo. Running scan_dir() on a directory will - with the __post_init__() feature of a Directory() object - automatically loop itself through the directory’s tree… More on that in the section below.

Directory Walking via post_init

A Directory() object unlike File() is mutable. I’m not sure if it’ll stay this way but for now it does technically mutate. When a Directory object is created as a result of scan_dir() it appends a class attribute of Directory.subDirs with the file path of a sub directory.

After initializing the object it runs __post_init__() which looks for string items in Directory.subDirs if it is a string it runs scan_dir() on that path. Then it creates a list of the new Directory() objects for the subDirs from the initial run. Once there’s no more strings it calls setattr on the object to replace the old string list of sub directories with Directory() objects.

def __post_init__(self):
        dir_objs = []
        for d in self.subDirs:
            dir_objs.append(scan_dir(d))
        self.__setattr__("subDirs", dir_objs)

Next Steps

Now I need to actually start the backup process. Moving files from the original directory location to the backup destination. I’m wondering if I should do the encryption before reaching the destination and if, I should compress the directory into an archive and encrypt that.

If I go that route then I could see the end result being a folder at the destination with an encrypted archive of the source directory with a manifest file being the dictionary dump of the root directory object (remember it nests sub directory Directory objects). A JSON dump of a test I ran of Directory object is below.

{
    "name": "admin",
    "path": "/home/unkwn1/admin",
    "subDirs": [
        {
            "name": "ecomm",
            "path": "/home/unkwn1/admin/ecomm",
            "subDirs": [
                {
                    "name": "docker-build",
                    "path": "/home/unkwn1/admin/ecomm/docker-build",
                    "subDirs": [
                        {
                            "name": "db",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/db",
                            "subDirs": [],
                            "files": []
                        },
                        {
                            "name": "html",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/html",
                            "subDirs": [],
                            "files": []
                        },
                        {
                            "name": "nginx",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/nginx",
                            "subDirs": [],
                            "files": [
                                {
                                    "name": "nginx.conf",
                                    "path": "/home/unkwn1/admin/ecomm/docker-build/nginx/nginx.conf",
                                    "size": 719,
                                    "owner": "1000/1000",
                                    "hash": "4cbd30c7bb67be77849efb5059e43328"
                                }
                            ]
                        }
                    ],
                    "files": [
                        {
                            "name": "db_root_pass.txt",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/db_root_pass.txt",
                            "size": 17,
                            "owner": "1000/1000",
                            "hash": "615420df60bec06da0e481e90aa499e8"
                        },
                        {
                            "name": "docker-compose2.yaml",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/docker-compose2.yaml",
                            "size": 1397,
                            "owner": "1000/1000",
                            "hash": "88c1f80fa0af424e957518fa8327a406"
                        },
                        {
                            "name": "db_name.txt",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/db_name.txt",
                            "size": 4,
                            "owner": "1000/1000",
                            "hash": "e9c71b19e7c9072312777bf0ecb91e62"
                        },
                        {
                            "name": "db_user.txt",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/db_user.txt",
                            "size": 8,
                            "owner": "1000/1000",
                            "hash": "0d1bd075a90d6d17856ea3ad78ca07d1"
                        },
                        {
                            "name": "docker-compose.yaml",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/docker-compose.yaml",
                            "size": 1397,
                            "owner": "1000/1000",
                            "hash": "88c1f80fa0af424e957518fa8327a406"
                        },
                        {
                            "name": "db_pass.txt",
                            "path": "/home/unkwn1/admin/ecomm/docker-build/db_pass.txt",
                            "size": 17,
                            "owner": "1000/1000",
                            "hash": "fd2ad0824af0889696c26a6a29edcc88"
                        }
                    ]
                }
            ],
            "files": [
                {
                    "name": "ecommerbal_container.tar",
                    "path": "/home/unkwn1/admin/ecomm/ecommerbal_container.tar",
                    "size": 1169688064,
                    "owner": "1000/1000",
                    "hash": "22efd7c27105f0ff5c7409f0024d7056"
                },
                {
                    "name": "backup_2022-05-07-0915_GTA_Herbal_92278c2b63ec-db",
                    "path": "/home/unkwn1/admin/ecomm/backup_2022-05-07-0915_GTA_Herbal_92278c2b63ec-db",
                    "size": 77826952,
                    "owner": "1000/1000",
                    "hash": "4903bd5c8fe01f3187d47e92e8d66a78"
                }
            ]
        },
        {
            "name": "cluster",
            "path": "/home/unkwn1/admin/cluster",
            "subDirs": [
                {
                    "name": "apps",
                    "path": "/home/unkwn1/admin/cluster/apps",
                    "subDirs": [
                        {
                            "name": "dashboard",
                            "path": "/home/unkwn1/admin/cluster/apps/dashboard",
                            "subDirs": [],
                            "files": [
                                {
                                    "name": "dashboard-admin.yaml",
                                    "path": "/home/unkwn1/admin/cluster/apps/dashboard/dashboard-admin.yaml",
                                    "size": 361,
                                    "owner": "1000/1000",
                                    "hash": "fad2bedae8cbd578bbfb5ec6e42e8aa9"
                                },
                                {
                                    "name": "dashboard-default.yaml",
                                    "path": "/home/unkwn1/admin/cluster/apps/dashboard/dashboard-default.yaml",
                                    "size": 7621,
                                    "owner": "1000/1000",
                                    "hash": "fc132ffdd61a0ac56c8cebf6678cbd0d"
                                },
                                {
                                    "name": "unkwn1token.txt",
                                    "path": "/home/unkwn1/admin/cluster/apps/dashboard/unkwn1token.txt",
                                    "size": 866,
                                    "owner": "1000/1000",
                                    "hash": "bb9f830747451c39f81a13b9197a3cbb"
                                }
                            ]
                        },
                        {
                            "name": "nextcloud",
                            "path": "/home/unkwn1/admin/cluster/apps/nextcloud",
                            "subDirs": [],
                            "files": []
                        }
                    ],
                    "files": []
                }
            ],
            "files": [
                {
                    "name": "install-krew.sh",
                    "path": "/home/unkwn1/admin/cluster/install-krew.sh",
                    "size": 570,
                    "owner": "1000/1000",
                    "hash": "171222a8f69cc8a4e50a79589d30a1be"
                }
            ]
        },
        {
            "name": "localhost",
            "path": "/home/unkwn1/admin/localhost",
            "subDirs": [],
            "files": [
                {
                    "name": "docker-compose.yml",
                    "path": "/home/unkwn1/admin/localhost/docker-compose.yml",
                    "size": 300,
                    "owner": "1000/1000",
                    "hash": "799a0b1416212872448a2e3c20772581"
                }
            ]
        }
    ],
    "files": [
        {
            "name": "etcd.service",
            "path": "/home/unkwn1/admin/etcd.service",
            "size": 972,
            "owner": "1000/1000",
            "hash": "4f7732298fb9b0e5966065d92c9d3e2c"
        }
    ]
}

Posts in this series