pickleDB is an open source key-value store using Python's json module.

Download badge

pickleDB

pickleDB is lightweight, fast, and simple database based on the json module. And it's BSD licensed!

pickleDB is Fun

>>> import pickledb

>>> db = pickledb.load('test.db', False)

>>> db.set('key', 'value')

>>> db.get('key')
'value'

>>> db.dump()
True

Easy to Install

$ pip install pickledb

Links

Latest Release Notes (version: 0.9)

  • rem(key) now returns False instead of raising an exception (0.9dev)
  • Change lrem(name) to lremlist(name) (0.9)
  • Add lremvalue(name, value) (0.9)
  • Add load() option to use sigterm handler or not (0.9)
  • All keys must now be strings (0.8)
  • All names for lists must now be strings (0.8)
  • All names for dicts must now be strings (0.8)
  • The get(key) function now returns False instead of None if there is no key (0.8)
  • Switched to Python's built in json module from simplejson (0.8.1)
Comments
  • Problem with reloading an already created db. TypeError: the JSON object must be str, not 'bytes'

    Problem with reloading an already created db. TypeError: the JSON object must be str, not 'bytes'

    import pickledb
    db = pickledb.load('example.db', False)
    db.set('key', 'value')
    db.dump()
    db = pickledb.load('example.db', False)
    

    Traceback (most recent call last): File "", line 1, in File "C:\Users\Documents\Python Projects\garbage-chatter\venv\lib\site-packages\pickledb.py", line 39, in load return pickledb(location, auto_dump) File "C:\Users\Documents\Python Projects\garbage-chatter\venv\lib\site-packages\pickledb.py", line 49, in init self.load(location, auto_dump) File "C:\Users\Documents\Python Projects\garbage-chatter\venv\lib\site-packages\pickledb.py", line 79, in load self.loaddb() File "C:\Users\Documents\Python Projects\garbage-chatter\venv\lib\site-packages\pickledb.py", line 96, in loaddb self.db = json.load(open(self.loco, 'rb')) File "C:\Users\AppData\Local\Programs\Python\Python35\Lib\json_init.py", line 268, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "C:\Users\AppData\Local\Programs\Python\Python35\Lib\json_init.py", line 312, in loads s.class.name)) TypeError: the JSON object must be str, not 'bytes'

  • source code changes key type rendering key lookups in failure

    source code changes key type rendering key lookups in failure

    as diagnosed by maintainers here; https://github.com/patx/pickledb/issues/37

    this shows the user is supplying an int and the framework changes the type to string.

    JSON supports integer tyes as values and talking about JSON β€œvalues” confuses this discussion on a key/value pair database where the key type is being discovered, not the value.

  • Database file corruption caused by SIGTERM during dump() execution

    Database file corruption caused by SIGTERM during dump() execution

    When a SIGTERM signal (e.g. by hitting Ctrl-C in terminal) is received during dump() execution, the exit is performed immediately which causes corruption of the pickledb database file. This can be very annoying as it might mean the loss of critical data.

    Upon execution of load() after a restart, I get this this error:

    Traceback (most recent call last):
      File "test.py", line 22, in <module>
        main()
      File "test.py", line 14, in main
        db = get_pickledb()
      File "test.py", line 9, in get_pickledb
        db = pickledb.load(DB_FILEPATH, False)
      File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 34, in load
        return pickledb(location, option)
      File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 41, in __init__
        self.load(location, option)
      File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 49, in load
        self._loaddb()
      File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 186, in _loaddb
        self.db = json.load(open(self.loco, 'rb'))
      File "/usr/lib/python2.7/json/__init__.py", line 290, in load
        **kw)
      File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
        raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded
    

    My pull request will fix this: https://github.com/patx/pickledb/pull/17

  • db.lremvalue(key, value) doesn't dump to file when auto_dump is True.

    db.lremvalue(key, value) doesn't dump to file when auto_dump is True.

    import pickledb
    
    db = pickledb.load('database.sql', True)
    db.set('key', ['one', 'two', 'three'])
    
    db.ladd('key', 'four')
    
    db.lremvalue('key', 'four') # You need to use db.dump() as a workaround
    # db.dump()
    
  • Won't work for large files

    Won't work for large files

    To anyone who might have stumbled upon this package.. as far as my understanding goes, pickleDB won't work for large dictionaries, since it keeps the entire dictionary in RAM (or loads the entire db-file at once using simplejson). So, if you are looking to work with very large disk-based dictionaries, you might be better off using the Sqlite database, and writing a dictionary like wrapper on top.

    Also, the Shelve module seems to work nicely for large dictionaries. However, it has the downside that it uses Pickle internally, and thus, the Database files are not directly readable.

    Please correct me if I am wrong. Thanks!

  • TypeError when executing dump function

    TypeError when executing dump function

    There seems to be an issue with the example provided by pickledb itself. When calling the dump function with Python 3, there is a TypeError produced by the code.

    Here's what was inputted in a test file:

    import pickledb
    db = pickledb.load("test.db", False)
    db.set("key", "value")
    print(db.get("key"))
    db.dump()
    

    Traceback:

    Traceback (most recent call last):
      File "tests.py", line 6, in <module>
        db.dump()
      File "/Users/shreydesai/GitHub/pickledb/pickledb.py", line 56, in dump
        self._dumpdb(True)
      File "/Users/shreydesai/GitHub/pickledb/pickledb.py", line 201, in _dumpdb
        simplejson.dump(self.db, open(self.loco, 'wb'))
      File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/
    simplejson/__init__.py", line 277, in dump
        fp.write(chunk)
    
    TypeError: 'str' does not support the buffer interface
    
  • rest web api for pickle

    rest web api for pickle

    I am thinking to write a web api wrapper around pickledb so that it can be accessed over http among multiple servers example : http://someurl:8080/set/a post : value

    http://someurl:8080/get/a returns its value in JSOiN format

    So I wanted to ask community its worth the effort or not.

  • Read-only file system error

    Read-only file system error

    Full error message:

    File "**hidden**/Functions/Main/m.py", line 32, in m
        db.dump()
    File "/opt/virtualenvs/python3/lib/python3.8/site-packages/pickledb.py", line 92, in dump
        json.dump(self.db, open(self.loco, 'wt'))
    OSError: [Errno 30] Read-only file system: '../../database.txt
    

    Code used: db = pickledb.load('../../database.txt', False)

  • DB file not created

    DB file not created

    I'm running a simple example, and pickledb seems to successfully set the value of key1 and prints it right after setting it. However, the pickle.db file is not created and attempting to get the value fails on the next run.

    import pickledb
    
    db = pickledb.load('pickle.db', False)
    
    db.set('key1', 'value')
    
    print(db.get('key1'))
    
    $ python3 pickledb_test.py
    value
    
    $ ls
    pickledb_test.py
    

    Using Python 3.8.5 and pickleDB 0.9.2.

    Edit: This was fixed by enabling auto_dump.

    pickledb.load('pickle.db', True)
    
  • Switching to OrderedDict

    Switching to OrderedDict

    Hi, thank you for reading this. What do you think about this idea to switch from an ordinary dictionary to the OrderedDict? I find that most of the times in my projects I need data to be sorted by keys so I have to do it manually. I thought maybe I'm not the only one and we can all benefit from making it a built-in feature? :) OrderedDict is located in the "collections" std-lib module.

  • Ability for the user to supply encryption function

    Ability for the user to supply encryption function

    It would be awesome to have optional encryption and decryption function supplied by the user so that they can encrypt the data before writing to file or decrypt using the decryptor function before reading from given file.

  • Getting JSONDecodeError

    Getting JSONDecodeError

    Hello,

    I'm getting JSON errors, and I'm not sure if I'm doing something wrong, or if its a bug.

    I'm getting this error on 3 different machines (2 Ubuntu, 1 Windows). One Ubuntu and Windows use python3.9, the other Ubuntu python3.8. All PickleDB's modules have been installed today via pip.

    Code:

    #/usr/bin/python3
    import pickledb
    
    db = pickledb.load('pik.db', False)
    db.set('foo', 'bar')
    
    print(db.get('foo'))
    

    When I run it:

    [email protected]:~/pik$ python3 pik.py 
    Traceback (most recent call last):
      File "pik.py", line 5, in <module>
        db = pickledb.load('pik.db', False)
      File "/home/ubuntu/.local/lib/python3.8/site-packages/pickledb.py", line 43, in load
        return PickleDB(location, auto_dump, sig)
      File "/home/ubuntu/.local/lib/python3.8/site-packages/pickledb.py", line 54, in __init__
        self.load(location, auto_dump)
      File "/home/ubuntu/.local/lib/python3.8/site-packages/pickledb.py", line 85, in load
        self._loaddb()
      File "/home/ubuntu/.local/lib/python3.8/site-packages/pickledb.py", line 102, in _loaddb
        self.db = json.load(open(self.loco, 'rt'))
      File "/usr/lib/python3.8/json/__init__.py", line 293, in load
        return loads(fp.read(),
      File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    

    Like to hear from you guys.

  • Ensure atomicity during database dump

    Ensure atomicity during database dump

    In order to ensure atomicity, dump the db into a temporary file before moving to the correct location. This will avoid creating empty file, mostly when process catch a SIGTERM signal.

    Also, clean up the extra dump done just before doing the same via a thread. This should reduce write overhead.

  • why json.dump data twice?

    why json.dump data twice?

    In the pickledb.py dump function(code line 90), the json.dump() func dumps data the first time, then the dthread dumps data second time.

    It seems a little confusing. Is that a bug or does anybody know why?

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database.

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database. This database is completely free* ??

Nov 25, 2022
LightDB is a lightweight JSON Database for Python

LightDB What is this? LightDB is a lightweight JSON Database for Python that allows you to quickly and easily write data to a file Installing pip3 ins

Oct 1, 2022
A Simple , ☁️ Lightweight , πŸ’ͺ Efficent JSON based database for 🐍 Python.
A Simple , ☁️ Lightweight ,  πŸ’ͺ Efficent JSON based database for 🐍 Python.

A Simple, Lightweight, Efficent JSON based DataBase for Python The current stable version is v1.6.1 pip install pysondb==1.6.1 Support the project her

Nov 27, 2022
securedb is a fast and lightweight Python framework to easily interact with JSON-based encrypted databases.

securedb securedb is a Python framework that lets you work with encrypted JSON databases. Features: newkey() to generate an encryption key write(key,

Nov 23, 2022
Tiny local JSON database for Python.
Tiny local JSON database for Python.

Pylowdb Simple to use local JSON database ?? # This is pure python, not specific to pylowdb ;) db.data['posts'] = ({ 'id': 1, 'title': 'pylowdb is awe

Jan 26, 2022
ClutterDB - Extremely simple JSON database made for infrequent changes which behaves like a dict

extremely simple JSON database made for infrequent changes which behaves like a dict this was made for ClutterBot

Jan 12, 2022
Shelf DB is a tiny document database for Python to stores documents or JSON-like data
Shelf DB is a tiny document database for Python to stores documents or JSON-like data

Shelf DB Introduction Shelf DB is a tiny document database for Python to stores documents or JSON-like data. Get it $ pip install shelfdb shelfquery S

Nov 3, 2022
Simpledb-py: Simple JSON database

Simpledb-py: Simple JSON database

Feb 9, 2022
Simple json type database for python3

What it is? Simple json type database for python3! What about speed? The speed is great! All data is stored in RAM until saved. How to install? pip in

Feb 11, 2022
Tools for analyzing Git history using SQLite

git-history Tools for analyzing Git history using SQLite Installation Install this tool using pip: $ pip install git-history Usage This tool can be r

Nov 29, 2022
Oh-My-PickleDB is an open source key-value store using Python's json module.
Oh-My-PickleDB is an open source key-value store using Python's json module.

OH-MY-PICKLEDB oh-my-pickleDB is a lightweight, fast, and intuitive data manager written in python ?? Table of Contents About Getting Started Deployme

Feb 20, 2022
A Simple Key-Value Data-store written in Python

mercury-db This is a File Based Key-Value Datastore that supports basic CRUD (Create, Read, Update, Delete) operations developed using Python. The dat

Jan 9, 2022
K2HASH Python library - NoSQL Key Value Store(KVS) library
K2HASH Python library - NoSQL Key Value Store(KVS) library

k2hash_python Overview k2hash_python is an official python driver for k2hash. Install Firstly you must install the k2hash shared library: curl -o- htt

Oct 19, 2022
CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, written in Python.
CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, written in Python.

CaskDB - Disk based Log Structured Hash Table Store CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, w

Nov 18, 2022
A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.
A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics, originally intended for keyboard layout optimization.

Nov 30, 2022
Random JSON Key:Pair Json Generator

Random JSON Key:Value Pair Generator This simple script take an engish dictionary of words and and makes random key value pairs. The dictionary has ap

Oct 14, 2021
AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database.

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database. This database is completely free* ??

Nov 25, 2022
🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.
🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.

A hyper-fast, safe Python module to read and write JSON data. Works as a drop-in replacement for Python's built-in json module. This is alpha software

Dec 6, 2022
Key Logger - Key Logger using Python

Key_Logger Key Logger using Python This is the basic Keylogger that i have made

Jan 15, 2022
Segcache: a memory-efficient and scalable in-memory key-value cache for small objects

Segcache: a memory-efficient and scalable in-memory key-value cache for small objects This repo contains the code of Segcache described in the followi

Nov 4, 2022