Silky smooth profiling for Django

Silk

GitHub Actions GitHub Actions PyPI Download PyPI Python Versions Jazzband

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection:

SECURITY NOTE: Because Silk stores all HTTP requests into the database in plain text, it will store the request's sensitive information into the database in plain text (e.g. users' passwords!). This is a massive security concern. An issue has been created for this here.

Contents

Requirements

Silk has been tested with:

  • Django: 2.2, 3.0, 3.1
  • Python: 3.6, 3.7, 3.8, 3.9

Installation

Via pip into a virtualenv:

pip install django-silk

In settings.py add the following:

MIDDLEWARE = [
    ...
    'silk.middleware.SilkyMiddleware',
    ...
]

INSTALLED_APPS = (
    ...
    'silk'
)

Note: The middleware placement is sensitive. If the middleware before silk.middleware.SilkyMiddleware returns from process_request then SilkyMiddleware will never get the chance to execute. Therefore you must ensure that any middleware placed before never returns anything from process_request. See the django docs for more information on this.

Note: If you are using django.middleware.gzip.GZipMiddleware, place that before silk.middleware.SilkyMiddleware, otherwise you will get an encoding error.

If you want to use custom middleware, for example you developed the subclass of silk.middleware.SilkyMiddleware, so you can use this combination of settings:

# Specify the path where is the custom middleware placed
SILKY_MIDDLEWARE_CLASS = 'path.to.your.middleware.MyCustomSilkyMiddleware'

# Use this variable in list of middleware
MIDDLEWARE = [
    ...
    SILKY_MIDDLEWARE_CLASS,
    ...
]

To enable access to the user interface add the following to your urls.py:

urlpatterns += [url(r'^silk/', include('silk.urls', namespace='silk'))]

before running migrate:

python manage.py makemigrations

python manage.py migrate

python manage.py collectstatic

Silk will automatically begin interception of requests and you can proceed to add profiling if required. The UI can be reached at /silk/

Alternative Installation

Via github tags:

pip install https://github.com/jazzband/silk/archive/<version>.tar.gz

You can install from master using the following, but please be aware that the version in master may not be working for all versions specified in requirements

pip install -e git+https://github.com/jazzband/django-silk.git#egg=silk

Features

Silk primarily consists of:

  • Middleware for intercepting Requests/Responses
  • A wrapper around SQL execution for profiling of database queries
  • A context manager/decorator for profiling blocks of code and functions either manually or dynamically.
  • A user interface for inspection and visualisation of the above.

Request Inspection

The Silk middleware intercepts and stores requests and responses in the configured database. These requests can then be filtered and inspecting using Silk's UI through the request overview:

It records things like:

  • Time taken
  • Num. queries
  • Time spent on queries
  • Request/Response headers
  • Request/Response bodies

and so on.

Further details on each request are also available by clicking the relevant request:

SQL Inspection

Silk also intercepts SQL queries that are generated by each request. We can get a summary on things like the tables involved, number of joins and execution time (the table can be sorted by clicking on a column header):

Before diving into the stack trace to figure out where this request is coming from:

Profiling

Turn on the SILKY_PYTHON_PROFILER setting to use Python's built-in cProfile profiler. Each request will be separately profiled and the profiler's output will be available on the request's Profiling page in the Silk UI.

SILKY_PYTHON_PROFILER = True

If you would like to also generate a binary .prof file set the following:

SILKY_PYTHON_PROFILER_BINARY = True

When enabled, a graph visualisation generated using gprof2dot and viz.js is shown in the profile detail page:

A custom storage class can be used for the saved generated binary .prof files:

SILKY_STORAGE_CLASS = 'path.to.StorageClass'

The default storage class is silk.storage.ProfilerResultStorage, and when using that you can specify a path of your choosing. You must ensure the specified directory exists.

# If this is not set, MEDIA_ROOT will be used.
SILKY_PYTHON_PROFILER_RESULT_PATH = '/path/to/profiles/'

A download button will become available with a binary .prof file for every request. This file can be used for further analysis using snakeviz or other cProfile tools

Silk can also be used to profile specific blocks of code/functions. It provides a decorator and a context manager for this purpose.

For example:

from silk.profiling.profiler import silk_profile


@silk_profile(name='View Blog Post')
def post(request, post_id):
    p = Post.objects.get(pk=post_id)
    return render_to_response('post.html', {
        'post': p
    })

Whenever a blog post is viewed we get an entry within the Silk UI:

Silk profiling not only provides execution time, but also collects SQL queries executed within the block in the same fashion as with requests:

Decorator

The silk decorator can be applied to both functions and methods

from silk.profiling.profiler import silk_profile


# Profile a view function
@silk_profile(name='View Blog Post')
def post(request, post_id):
    p = Post.objects.get(pk=post_id)
    return render_to_response('post.html', {
        'post': p
    })


# Profile a method in a view class
class MyView(View):
    @silk_profile(name='View Blog Post')
    def get(self, request):
        p = Post.objects.get(pk=post_id)
        return render_to_response('post.html', {
            'post': p
        })

Context Manager

Using a context manager means we can add additional context to the name which can be useful for narrowing down slowness to particular database records.

def post(request, post_id):
    with silk_profile(name='View Blog Post #%d' % self.pk):
        p = Post.objects.get(pk=post_id)
        return render_to_response('post.html', {
            'post': p
        })

Dynamic Profiling

One of Silk's more interesting features is dynamic profiling. If for example we wanted to profile a function in a dependency to which we only have read-only access (e.g. system python libraries owned by root) we can add the following to settings.py to apply a decorator at runtime:

SILKY_DYNAMIC_PROFILING = [{
    'module': 'path.to.module',
    'function': 'MyClass.bar'
}]

which is roughly equivalent to:

class MyClass(object):
    @silk_profile()
    def bar(self):
        pass

The below summarizes the possibilities:

"""
Dynamic function decorator
"""

SILKY_DYNAMIC_PROFILING = [{
    'module': 'path.to.module',
    'function': 'foo'
}]

# ... is roughly equivalent to
@silk_profile()
def foo():
    pass

"""
Dynamic method decorator
"""

SILKY_DYNAMIC_PROFILING = [{
    'module': 'path.to.module',
    'function': 'MyClass.bar'
}]

# ... is roughly equivalent to
class MyClass(object):

    @silk_profile()
    def bar(self):
        pass

"""
Dynamic code block profiling
"""

SILKY_DYNAMIC_PROFILING = [{
    'module': 'path.to.module',
    'function': 'foo',
    # Line numbers are relative to the function as opposed to the file in which it resides
    'start_line': 1,
    'end_line': 2,
    'name': 'Slow Foo'
}]

# ... is roughly equivalent to
def foo():
    with silk_profile(name='Slow Foo'):
        print (1)
        print (2)
    print(3)
    print(4)

Note that dynamic profiling behaves in a similar fashion to that of the python mock framework in that we modify the function in-place e.g:

""" my.module """
from another.module import foo

# ...do some stuff
foo()
# ...do some other stuff

,we would profile foo by dynamically decorating my.module.foo as opposed to another.module.foo:

SILKY_DYNAMIC_PROFILING = [{
    'module': 'my.module',
    'function': 'foo'
}]

If we were to apply the dynamic profile to the functions source module another.module.foo after it has already been imported, no profiling would be triggered.

Custom Logic for Profiling

Sometimes you may want to dynamically control when the profiler runs. You can write your own logic for when to enable the profiler. To do this add the following to your settings.py:

This setting is mutually exclusive with SILKY_PYTHON_PROFILER and will be used over it if present. It will work with SILKY_DYNAMIC_PROFILING.

def my_custom_logic(request):
    return 'profile_requests' in request.session

SILKY_PYTHON_PROFILER_FUNC = my_custom_logic # profile only session has recording enabled.

You can also use a lambda.

# profile only session has recording enabled.
SILKY_PYTHON_PROFILER_FUNC = lambda request: 'profile_requests' in request.session

Code Generation

Silk currently generates two bits of code per request:

Both are intended for use in replaying the request. The curl command can be used to replay via command-line and the python code can be used within a Django unit test or simply as a standalone script.

Configuration

Authentication/Authorisation

By default anybody can access the Silk user interface by heading to /silk/. To enable your Django auth backend place the following in settings.py:

SILKY_AUTHENTICATION = True  # User must login
SILKY_AUTHORISATION = True  # User must have permissions

If SILKY_AUTHORISATION is True, by default Silk will only authorise users with is_staff attribute set to True.

You can customise this using the following in settings.py:

def my_custom_perms(user):
    return user.is_allowed_to_use_silk

SILKY_PERMISSIONS = my_custom_perms

You can also use a lambda.

SILKY_PERMISSIONS = lambda user: user.is_superuser

Request/Response bodies

By default, Silk will save down the request and response bodies for each request for future viewing no matter how large. If Silk is used in production under heavy volume with large bodies this can have a huge impact on space/time performance. This behaviour can be configured with the following options:

SILKY_MAX_REQUEST_BODY_SIZE = -1  # Silk takes anything <0 as no limit
SILKY_MAX_RESPONSE_BODY_SIZE = 1024  # If response body>1024 bytes, ignore

Meta-Profiling

Sometimes it is useful to be able to see what effect Silk is having on the request/response time. To do this add the following to your settings.py:

SILKY_META = True

Silk will then record how long it takes to save everything down to the database at the end of each request:

Note that in the above screenshot, this means that the request took 29ms (22ms from Django and 7ms from Silk)

Recording a Fraction of Requests

On high-load sites it may be helpful to only record a fraction of the requests that are made. To do this add the following to your settings.py:

Note: This setting is mutually exclusive with SILKY_INTERCEPT_FUNC.

SILKY_INTERCEPT_PERCENT = 50 # log only 50% of requests

Custom Logic for Recording Requests

On high-load sites it may also be helpful to write your own logic for when to intercept requests. To do this add the following to your settings.py:

Note: This setting is mutually exclusive with SILKY_INTERCEPT_PERCENT.

def my_custom_logic(request):
    return 'record_requests' in request.session

SILKY_INTERCEPT_FUNC = my_custom_logic # log only session has recording enabled.

You can also use a lambda.

# log only session has recording enabled.
SILKY_INTERCEPT_FUNC = lambda request: 'record_requests' in request.session

Limiting request/response data

To make sure silky garbage collects old request/response data, a config var can be set to limit the number of request/response rows it stores.

SILKY_MAX_RECORDED_REQUESTS = 10**4

The garbage collection is only run on a percentage of requests to reduce overhead. It can be adjusted with this config:

SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT = 10

Enable query analysis

To enable query analysis when supported by the dbms a config var can be set in order to execute queries with the analyze features.

SILKY_ANALYZE_QUERIES = True

Clearing logged data

A management command will wipe out all logged data:

python manage.py silk_clear_request_log

Contributing

Jazzband

This is a Jazzband project. By contributing you agree to abide by the Contributor Code of Conduct and follow the guidelines.

Development Environment

Silk features a project named project that can be used for silk development. It has the silk code symlinked so you can work on the sample project and on the silk package at the same time.

In order to setup local development you should first install all the dependencies for the test project. From the root of the project directory:

pip install -r requirements.txt

You will also need to install silk's dependencies. From the root of the git repository:

pip install -e .

At this point your virtual environment should have everything it needs to run both the sample project and silk successfully.

Before running, you must set the DB_ENGINE and DB_NAME environment variables:

export DB_ENGINE=sqlite3
export DB_NAME=db.sqlite3

For other combinations, check tox.ini.

Now from the root of the sample project apply the migrations

python manage.py migrate

Now from the root of the sample project directory start the django server

python manage.py runserver

Running the tests

cd project
./tests/test_migrations.sh
python manage.py test --noinput

Happy profiling!

Comments
  • IntegrityError: duplicate key value violates unique constraint

    IntegrityError: duplicate key value violates unique constraint "silk_response_request_id_key"

    After activating silk certain urls began erring with:

    IntegrityError: duplicate key value violates unique constraint "silk_response_request_id_key" DETAIL: Key (request_id)=(1166) already exists.

    Sentry stack trace: http://toolbox1.tedc.de:9000/bidev/esldj/group/131/

  • Support Django 3.2

    Support Django 3.2

    Currently, support for Django 3.2 is broken, as per ~~this issue.~~ (EDIT: the linked issue isn't related to Django 3.2, though support was broken before the PR I have submitted, and I now believe it is fixed)

    Therefore, I created this issue to track progress on fixing django-silk in Django 3.2

  • Create new release of silk

    Create new release of silk

    I think once the jazzband migration is complete, it would be a good time to make a new release of silk, either manually or by setting up automatic releases.

  • Prepare Release 4.0

    Prepare Release 4.0

    Since this is a major release as we are dropping python 2 support, we will move to 4.x instead of 3.x.

    • [x] Check the support with Django 3.0
    • [x] Test django silk if its working fine.
    • [x] Prepare a change log
  • `apps.py` added with `default_auto_field` set

    `apps.py` added with `default_auto_field` set

    This is to avoid unnecessary migrations caused by default "id" field type change from AutoField -> BigAutoField in Django 3.2.

    new test added

    • testing if proper AppConfig class is loaded by Django
    • testing if package has no pending migrations and will not generate new migrations when running makemigrations on user-end.
  • Migrations not working w/ Django 1.6.11

    Migrations not working w/ Django 1.6.11

    Traceback (most recent call last):
      File "manage.py", line 13, in <module>
        execute_from_command_line(sys.argv)
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
        utility.execute()
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
        self.execute(*args, **options.__dict__)
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
        output = self.handle(*args, **options)
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
        ignore_ghosts = ignore_ghosts,
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/__init__.py", line 173, in migrate_app
        Migrations.calculate_dependencies()
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/base.py", line 229, in calculate_dependencies
        migration.calculate_dependencies()
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/base.py", line 363, in calculate_dependencies
        for migration in self._get_dependency_objects("depends_on"):
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/base.py", line 343, in _get_dependency_objects
        for app, name in getattr(self.migration_class(), attrname, []):
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/base.py", line 315, in migration_class
        return self.migration().Migration
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/utils/__init__.py", line 62, in method
        value = function(self)
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/base.py", line 304, in migration
        raise exceptions.UnknownMigration(self, sys.exc_info())
    south.exceptions.UnknownMigration: Migration 'silk:0001_initial' probably doesn't exist.
    Traceback (most recent call last):
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/south/migration/base.py", line 302, in migration
        migration = __import__(full_name, {}, {}, ['Migration'])
      File "/Users/matthew/.virtualenvs/nauman/lib/python2.7/site-packages/silk/migrations/0001_initial.py", line 4, in <module>
        from django.db import models, migrations
    ImportError: cannot import name migrations
    

    The solution from https://github.com/django-silk/silk/issues/64 did not fix this.

  • Profiling is not working

    Profiling is not working

    I'm trying to profile a api of this project. I had followed the instructions of the readme of django-silk, but the profiling page was empty, though i could see files generated under profiles directory which i configured.

    The project's address is https://github.com/bluven/py-perf, it's a simple project, the method i profiled is api/views/UserViewSet.list

    settings.py

    INSTALLED_APPS = [
        'django.contrib.admin',
        'django.contrib.auth',
        'django.contrib.contenttypes',
        'django.contrib.sessions',
        'django.contrib.messages',
        'django.contrib.staticfiles',
        'silk'
    ]
    
    MIDDLEWARE = [
        'silk.middleware.SilkyMiddleware',
        'django.middleware.cache.UpdateCacheMiddleware',
        'django.middleware.security.SecurityMiddleware',
        'django.contrib.sessions.middleware.SessionMiddleware',
        'django.middleware.common.CommonMiddleware',
        'django.middleware.csrf.CsrfViewMiddleware',
        'django.contrib.auth.middleware.AuthenticationMiddleware',
        'django.contrib.messages.middleware.MessageMiddleware',
        'django.middleware.clickjacking.XFrameOptionsMiddleware',
        'django.middleware.cache.FetchFromCacheMiddleware',
    ]
    
    SILKY_PYTHON_PROFILER = True
    SILKY_PYTHON_PROFILER_BINARY = True
    SILKY_PYTHON_PROFILER_RESULT_PATH = os.path.join(BASE_DIR, "profiles")
    

    api/views.py

    class UserViewSet(ModelViewSet):
        queryset = User.objects.all()
        serializer_class = UserSerializer
        lookup_field = 'username'
    
        @silk_profile()
        def list(self, request, *args, **kwargs):
            print('*')
            return super(UserViewSet, self).list(request, *args, **kwargs)
    
    

    Thanks in advance.

  • NoReverseMatch: Reverse for 'summary' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []

    NoReverseMatch: Reverse for 'summary' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []

    I am getting NoReverseMatch error. I tried placing middleware at top but its giving same error.

    These are my middleware classes:

    MIDDLEWARE_CLASSES = (
        'django.contrib.sessions.middleware.SessionMiddleware',
        'subdomains.middleware.SubdomainURLRoutingMiddleware',
        'corsheaders.middleware.CorsMiddleware',
        'django.middleware.common.CommonMiddleware',
        'django.middleware.csrf.CsrfViewMiddleware',
        'django.contrib.auth.middleware.AuthenticationMiddleware',
        'django.contrib.messages.middleware.MessageMiddleware',
        'django.middleware.clickjacking.XFrameOptionsMiddleware',
        'social.apps.django_app.middleware.SocialAuthExceptionMiddleware',
        'django_mobile.middleware.MobileDetectionMiddleware',
        'django_mobile.middleware.SetFlavourMiddleware',
    )
    

    and its the traceback-

    Traceback (most recent call last):
      File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
        self.result = application(self.environ, self.start_response)
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 64, in __call__
        return self.application(environ, start_response)
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 168, in __call__
        self.load_middleware()
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 44, in load_middleware
        mw_class = import_string(middleware_path)
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_string
        module = import_module(module_path)
      File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
        __import__(name)
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/silk/middleware.py", line 31, in <module>
        fpath = silky_reverse('summary')
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/silk/middleware.py", line 27, in silky_reverse
        r = reverse(name, *args, **kwargs)
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 551, in reverse
        return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))
      File "/home/ashish/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 468, in _reverse_with_prefix
        (lookup_view_s, args, kwargs, len(patterns), patterns))
    NoReverseMatch: Reverse for 'summary' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []
    [24/Aug/2015 14:15:14] "GET /silk HTTP/1.1" 500 59
    

    Is this error due to middleware placement or its some other issue ?

  • Django 2.0 support

    Django 2.0 support

    I have this error after update:

     File "/Users/izvr/Projects/bet/env/lib/python3.6/site-packages/silk/models.py", line 135, in Response
        request = OneToOneField(Request, related_name='response', db_index=True)
    TypeError: __init__() missing 1 required positional argument: 'on_delete'
    
  • DB Deadlock when stress testing with silk

    DB Deadlock when stress testing with silk

    I got a deadlock error when testing with the latest version of silk (1.1.0).

    Here is the DB output:

            Process 149: DELETE FROM "silk_response" WHERE "silk_response"."request_id" IN ('629f685a-26dc-41ca-ba9d-cc8af1998ef9', '8336effc-217c-47a7-8cc2-6944b8aede7f', 'aa12852b-5f5b-40fa-a647-cc3ddd2ef4ba', '2b2d5e84-e595-4ccf-b06c-78db181b4f24', '3f94c00d-f0c3-4021-8def-703e363e4f8f', 'e88b9162-0455-4e78-ba7b-029694b493d8')
            Process 150: DELETE FROM "silk_response" WHERE "silk_response"."request_id" IN ('629f685a-26dc-41ca-ba9d-cc8af1998ef9', '9ad245bb-4b6a-49cf-b32c-1a547ee8c11f', '93c50572-f6ec-416c-a9d6-42798f1631dd', '1ffd6a86-f93e-48f8-a593-86184e26c417', '3f94c00d-f0c3-4021-8def-703e363e4f8f')
            Process 151: DELETE FROM "silk_response" WHERE "silk_response"."request_id" IN ('629f685a-26dc-41ca-ba9d-cc8af1998ef9', '9ad245bb-4b6a-49cf-b32c-1a547ee8c11f', 'aa12852b-5f5b-40fa-a647-cc3ddd2ef4ba', '8336effc-217c-47a7-8cc2-6944b8aede7f', '2b2d5e84-e595-4ccf-b06c-78db181b4f24', '93c50572-f6ec-416c-a9d6-42798f1631dd', '1ffd6a86-f93e-48f8-a593-86184e26c417', '3f94c00d-f0c3-4021-8def-703e363e4f8f', 'e88b9162-0455-4e78-ba7b-029694b493d8')
            Process 144: DELETE FROM "silk_response" WHERE "silk_response"."request_id" IN ('629f685a-26dc-41ca-ba9d-cc8af1998ef9', '9ad245bb-4b6a-49cf-b32c-1a547ee8c11f', 'aa12852b-5f5b-40fa-a647-cc3ddd2ef4ba', '8336effc-217c-47a7-8cc2-6944b8aede7f', '2b2d5e84-e595-4ccf-b06c-78db181b4f24', '93c50572-f6ec-416c-a9d6-42798f1631dd', '1ffd6a86-f93e-48f8-a593-86184e26c417', '3f94c00d-f0c3-4021-8def-703e363e4f8f', 'e88b9162-0455-4e78-ba7b-029694b493d8')
    2018-01-26 01:03:55.996 UTC [148] HINT:  See server log for query details.
    2018-01-26 01:03:55.996 UTC [148] STATEMENT:  DELETE FROM "silk_response" WHERE "silk_response"."request_id" IN ('629f685a-26dc-41ca-ba9d-cc8af1998ef9', '9ad245bb-4b6a-49cf-b32c-1a547ee8c11f', 'aa12852b-5f5b-40fa-a647-cc3ddd2ef4ba', '8336effc-217c-47a7-8cc2-6944b8aede7f', '2b2d5e84-e595-4ccf-b06c-78db181b4f24', '1ffd6a86-f93e-48f8-a593-86184e26c417', '3f94c00d-f0c3-4021-8def-703e363e4f8f', 'e88b9162-0455-4e78-ba7b-029694b493d8')
    2018-01-26 01:03:56.024 UTC [144] ERROR:  deadlock detected
    2018-01-26 01:03:56.024 UTC [144] DETAIL:  Process 144 waits for AccessExclusiveLock on tuple (443,8) of relation 16579 of database 12994; blocked by process 149.
            Process 149 waits for ShareLock on transaction 152027; blocked by process 150.
            Process 150 waits for AccessExclusiveLock on tuple (440,10) of relation 16579 of database 12994; blocked by process 151.
            Process 151 waits for ShareLock on transaction 152109; blocked by process 144.
    

    Output from pip freeze:

    amqp==2.2.2
    autopep8==1.3.4
    billiard==3.5.0.3
    blessings==1.6.1
    bpython==0.17
    cached-property==1.3.1
    celery==4.1.0
    certifi==2018.1.18
    chardet==3.0.4
    coreapi==2.3.3
    coreschema==0.0.4
    coverage==4.4.2
    curtsies==0.2.11
    Django==1.11.9
    django-auth-ldap==1.3.0
    django-cachalot==1.5.0
    django-extensions==1.9.9
    django-model-utils==3.1.1
    django-prettyjson==0.3.0
    django-rest-swagger==2.1.2
    django-silk==1.1.0
    django-url-filter==0.3.4
    djangorestframework==3.7.7
    drf-extensions==0.3.1
    drf-nested-routers==0.90.0
    enum-compat==0.0.2
    gevent==1.2.2
    gprof2dot==2016.10.13
    greenlet==0.4.12
    gunicorn==19.7.1
    idna==2.6
    itypes==1.1.0
    Jinja2==2.10
    kombu==4.1.0
    Markdown==2.6.11
    MarkupSafe==1.0
    mercury==1.0.0
    mock==2.0.0
    openapi-codec==1.3.2
    pbr==3.1.1
    psycopg2==2.7.3.2
    pycodestyle==2.3.1
    Pygments==2.2.0
    pyldap==2.4.45
    python-dateutil==2.6.1
    python-memcached==1.59
    pytz==2017.3
    requests==2.18.4
    simplejson==3.13.2
    six==1.11.0
    sqlparse==0.2.4
    standardjson==0.3.1
    stevedore==1.28.0
    typing==3.6.4
    uritemplate==3.0.0
    urllib3==1.22
    vine==1.1.4
    wcwidth==0.1.7
    Werkzeug==0.14.1
    

    Thanks -Selim

  • Quality of life improvements in Requests tab

    Quality of life improvements in Requests tab

    I'm using you're package quite often and wanted to give something back to this project 😄

    This PR focuses on improving experience on Request tab - one improvement per commit:

    • Add message when there are no requests to display: when there are no matches instead of displaying an empty page, let user know that there are no filter results
    • Add 'Clear all filters' button - when user fills in a few filters, clearing them is rather tedious task so I thought it would be nice to have a button to clear filters.
    • Add small margin for filter selects: I think this change makes filter section looks much better.

    This branch is based off 4.2.0 tag, please let me know if I need to rebase.

  • Run django-upgrade manually

    Run django-upgrade manually

    Debugging the failures in #620 , I ran django-upgrade manually to find what it was doing to files to cause the tests to fail. This PR contains the changes from running django-upgrade manually (and matches #620) except for the below change from django-upgrade which breaks tests and therefore is not included in this PR:

    diff --git a/silk/model_factory.py b/silk/model_factory.py
    index ba21fed..9a091c0 100644
    --- a/silk/model_factory.py
    +++ b/silk/model_factory.py
    @@ -74,7 +74,7 @@ class RequestModelFactory:
             self.request = request
    
         def content_type(self):
    -        content_type = self.request.META.get('CONTENT_TYPE', '')
    +        content_type = self.request.headers.get('content-type', '')
             return _parse_content_type(content_type)
    
         def encoded_headers(self):
    

    After this PR is landed, pre-commit CI can regenerate #620 and the auto-linter can continue to work. https://github.com/albertyw/django-silk/blob/master/.pre-commit-config.yaml#L54-L58 may need to be commented out though to prevent the above change from reappearing.

  • Allow to generate more informative profile file name

    Allow to generate more informative profile file name

    The aim of this pull request is to generate a more informative profile file name. This would allow us to identify the endpoint that generates a profile file without the need to open it.

    To avoid breaking any existent use case I set this behavior behind the setting SILKY_PYTHON_PROFILER_EXTENDED_FILE_NAME that by default is disabled.

    If you think we can change the generated filename without worry we can simply apply this change always without the need to introduce the new settings.

  • RecursionError

    RecursionError

    Hi folks,

    thanks a lot for this great tool!

    When use it in drf project got RecursionError after some time. More deeper analysis showed that some functions get called recursively more and more times and after standard recursion depth = 1000 we see error 500.

    Restarting of python manage.py runserver Resets this "counter".

    What can it be?

    2022-12-17 13:00:35,666-ERROR-catalog_logger::middleware|56:: [email protected] GET /api/catalogs/medicine_groups/ occurred exception
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 161, in wrapped_target
        result = target(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 159, in wrapped_target
        self._start_queries()
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 78, in _start_queries
        self._queries_before = self._query_identifiers_from_collector()
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 74, in _query_identifiers_from_collector
        return [x for x in DataCollector().queries]
      File "/usr/local/lib/python3.10/site-packages/silk/collector.py", line 68, in queries
        return self._get_objects(TYP_QUERIES)
      File "/usr/local/lib/python3.10/site-packages/silk/collector.py", line 75, in _get_objects
        objects = self.objects
      File "/usr/local/lib/python3.10/site-packages/silk/collector.py", line 64, in objects
        return getattr(self.local, 'objects', None)
    RecursionError: maximum recursion depth exceeded in comparison
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
        response = handler(request, *args, **kwargs)
      File "/opt/app/utils/mixins.py", line 74, in list
        return super().list(request, *args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 161, in wrapped_target
        result = target(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 161, in wrapped_target
        result = target(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 161, in wrapped_target
        result = target(*args, **kwargs)
      [Previous line repeated 941 more times]
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 168, in wrapped_target
        self._finalise_queries()
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 109, in _finalise_queries
        self._end_queries()
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 82, in _end_queries
        self._queries_after = self._query_identifiers_from_collector()
      File "/usr/local/lib/python3.10/site-packages/silk/profiling/profiler.py", line 74, in _query_identifiers_from_collector
        return [x for x in DataCollector().queries]
      File "/usr/local/lib/python3.10/site-packages/silk/collector.py", line 68, in queries
        return self._get_objects(TYP_QUERIES)
      File "/usr/local/lib/python3.10/site-packages/silk/collector.py", line 75, in _get_objects
        objects = self.objects
      File "/usr/local/lib/python3.10/site-packages/silk/collector.py", line 64, in objects
        return getattr(self.local, 'objects', None)
    RecursionError: maximum recursion depth exceeded in comparison
    

    Can dynamic profiling of generic DRF views mixins be involdev?

    SILKY_DYNAMIC_PROFILING = [
        {
            'module': 'rest_framework.mixins.RetrieveModelMixin',
            'function': 'retrieve'
        },
        {
            'module': 'rest_framework.mixins.CreateModelMixin',
            'function': 'create'
        },
        {
            'module': 'rest_framework.mixins.ListModelMixin',
            'function': 'list'
        },
        {
            'module': 'rest_framework.mixins.UpdateModelMixin',
            'function': 'update'
        },
        {
            'module': 'rest_framework.mixins.DestroyModelMixin',
            'function': 'destroy'
        },
    ]
    
    
  • Allow to delete some requests

    Allow to delete some requests

    In development, I let silk live in the background and I often spawn its summary page to see page that takes the most time/queries.

    Now I sometimes use pdb on some requests so the time calculated for these is not relevant, and it would be nice to have a button to remove them from the DB so that the summary page can be updated.

Related tags
Mixer -- Is a fixtures replacement. Supported Django, Flask, SqlAlchemy and custom python objects.

The Mixer is a helper to generate instances of Django or SQLAlchemy models. It's useful for testing and fixture replacement. Fast and convenient test-

Dec 25, 2022
A Django plugin for pytest.

Welcome to pytest-django! pytest-django allows you to test your Django project/applications with the pytest testing tool. Quick start / tutorial Chang

Dec 31, 2022
Django test runner using nose

django-nose django-nose provides all the goodness of nose in your Django tests, like: Testing just your apps by default, not all the standard ones tha

Dec 15, 2022
Useful additions to Django's default TestCase
Useful additions to Django's default TestCase

django-test-plus Useful additions to Django's default TestCase from REVSYS Rationale Let's face it, writing tests isn't always fun. Part of the reason

Dec 22, 2022
Mixer -- Is a fixtures replacement. Supported Django, Flask, SqlAlchemy and custom python objects.

The Mixer is a helper to generate instances of Django or SQLAlchemy models. It's useful for testing and fixture replacement. Fast and convenient test-

Feb 4, 2021
Object factory for Django

Model Bakery: Smart fixtures for better tests Model Bakery offers you a smart way to create fixtures for testing in Django. With a simple and powerful

Jan 8, 2023
Test django schema and data migrations, including migrations' order and best practices.

django-test-migrations Features Allows to test django schema and data migrations Allows to test both forward and rollback migrations Allows to test th

Dec 27, 2022
Useful additions to Django's default TestCase
Useful additions to Django's default TestCase

django-test-plus Useful additions to Django's default TestCase from REVSYS Rationale Let's face it, writing tests isn't always fun. Part of the reason

Dec 22, 2022
A feature flipper for Django

README Django Waffle is (yet another) feature flipper for Django. You can define the conditions for which a flag should be active, and use it in a num

Jan 6, 2023
Silky smooth profiling for Django
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jan 1, 2023
Silky smooth profiling for Django
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jan 1, 2023
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper

Smooth ReLU in PyTorch Unofficial PyTorch reimplementation of the Smooth ReLU (SmeLU) activation function proposed in the paper Real World Large Scale

Jan 2, 2023
Dual languaged (rus+eng) tool for packing and unpacking archives of Silky Engine.
Dual languaged (rus+eng) tool for packing and unpacking archives of Silky Engine.

SilkyArcTool English Dual languaged (rus+eng) GUI tool for packing and unpacking archives of Silky Engine. It is not the same arc as used in Ai6WIN. I

Sep 15, 2022
Silk is a live profiling and inspection tool for the Django framework.
Silk is a live profiling and inspection tool for the Django framework.

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection:

Jan 2, 2023
This code extends the neural style transfer image processing technique to video by generating smooth transitions between several reference style images
This code extends the neural style transfer image processing technique to video by generating smooth transitions between several reference style images

Neural Style Transfer Transition Video Processing By Brycen Westgarth and Tristan Jogminas Description This code extends the neural style transfer ima

Jan 7, 2023
Tensorflow Implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE

SMU A Tensorflow Implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE arXiv https://arxiv.org/abs/211

Jan 18, 2022
A smooth and powerful Telegram Userbot made to make Telegram easier.
A smooth and powerful Telegram Userbot made to make Telegram easier.

| Xᴇɴᴏ Bᴏᴛ Is One Of The Fastest & Smoothest Bot On Telegram Based on Telethon|

Dec 1, 2021
(OLD REPO) Line-by-line profiling for Python - Current repo ->

line_profiler and kernprof line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either

Jan 6, 2023
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Jan 1, 2023
Line-by-line profiling for Python

line_profiler and kernprof NOTICE: This is the official line_profiler repository. The most recent version of line-profiler on pypi points to this repo

Dec 31, 2022