Add new model and API endpoints#

This guide aims to enable developers with little or no django experience to add django models and API endpoints to the project. Most code examples are followed by detailed explanations.

The developer will have exposure to the following skills in this document

  • python

  • django

  • django rest framework

  • relational database through the Django ORM (object-relational mapper)

  • data types

  • object-oriented concepts (object, inheritance, composition)

  • unit testing

  • API design

  • command line

This guide assumes the developer has followed the contributing doc and have forked and created a local branch to work on this. The development server would be already running in the background and will automatically apply the changes when we save the files.

We will choose the recurring_event issue as an example. Our goal is to create a database table and an API that a client can use to work with the data. The work is split into 3 testable components: the model, the admin site, and the API

Let’s start!

1. Data model#

TDD section

  1. Write the test

    We would like the model to store these data, and to return the name property in the str function.

    app/core/tests/test_models.py#
     1def test_recurring_event_model(project):
 2    from datetime import datetime
 3
 4    payload = {
 5        "name": "test event",
 6        "start_time": datetime(2023, 1, 1, 2, 34),
 7        "duration_in_min": 60,
 8        "video_conference_url": "https://zoom.com/mtg/1234",
 9        "additional_info": "long description",
10        "project": project,
11    }
12    recurring_event = RecurringEvent(**payload)
13    # recurring_event.save()
14    assert recurring_event.name == payload["name"]
15    assert recurring_event.start_time == payload["start_time"]
16    assert recurring_event.duration_in_min == payload["duration_in_min"]
17    assert recurring_event.video_conference_url == payload["video_conference_url"]
18    assert recurring_event.additional_info == payload["additional_info"]
19    assert recurring_event.project == payload["project"]
20    assert str(recurring_event) == payload["name"]

  2. See it fail

    run the following#
    ./scripts/test.sh

  3. Run it after implementing the model to make sure the code satisfies the test

1.1. Add the model#

Add the following to app/core/models.py

app/core/models.py#
 1class RecurringEvent(AbstractBaseModel):
 2    """
 3    Recurring Events
 4    """
 5
 6    name = models.CharField(max_length=255)
 7    start_time = models.TimeField("Start", null=True, blank=True)
 8    duration_in_min = models.IntegerField(null=True, blank=True)
 9    video_conference_url = models.URLField(blank=True)
10    additional_info = models.TextField(blank=True)
11
12    project = models.ForeignKey(Project, on_delete=models.CASCADE)
13    # location_id = models.ForeignKey("Location", on_delete=models.DO_NOTHING)
14    # event_type_id = models.ForeignKey("EventType", on_delete=models.DO_NOTHING)
15    # brigade_id = models.ForeignKey("Brigade", on_delete=models.DO_NOTHING)
16    # day_of_week = models.ForeignKey("DayOfWeek", on_delete=models.DO_NOTHING)
17    # must_roles = models.ManyToManyField("Role")
18    # should_roles = models.ManyToManyField("Role")
19    # could_roles = models.ManyToManyField("Role")
20    # frequency_id = models.ForeignKey("Frequency", on_delete=models.DO_NOTHING)
21
22    def __str__(self):
23        return f"{self.name}"

  1. We inherit all models from AbstractBaseModel, which provides a uuid primary key, created_at, and updated_at timestamps. In the Github issue, these fields might be called id, created, and updated. There’s no need to add those.

  2. Most fields should not be required. Text fields should be blank=True, data fields should be null=True.

  3. The data types in the github issue may be given in database column types such as INTEGER, VARCHAR, but we need to convert them into Django field types when defining the model.

  4. VARCHAR can be either CharField or TextField.

    1. CharField has a max_length, which makes it useful for finite length text data. We’re going default to giving them max_length=255 unless there’s a better value like max_length=2 for state abbreviation.

    2. TextField doesn’t have a maximum length, which makes it ideal for large text fields such as description.

  5. Try to add the relationships to non-existent models, but comment them out. Another developer will complete them when they go to implement those models.

  6. Always override the __str__ function to output something more meaningful than the default. It lets us do a quick test of the model by calling str([model]). It’s also useful for the admin site model list view.

1.2. Run migrations#

Run migrations to generate database migration files

./scripts/migrate.sh

1.3. Write a simple test#

Since we overrode the __str__ function, we need to write a test for it.

  1. Add a fixture for the model

    Fixtures are reusable code that can be used in multiple tests by declaring them as parameters of the test case. In this example, we show both defining a fixture (recurring_event) and using another fixture (project).

    Note: The conftest file is meant to hold shared test fixtures, among other things. The fixtures have directory scope.

    app/core/tests/conftest.py#
    1@pytest.fixture
2def recurring_event(project):
3    return RecurringEvent.objects.create(name="Test Recurring Event", project=project)

    1. We name the fixture after the model name (recurring_event).

    2. This model makes use of the project model as a foreign key relation, so we pass in the project fixture, which creates a project model.

    3. We create an object of the new model, passing in at least the required fields. In this case, we passed in enough arguments to use the __str__ method in a test.

  2. Add a test case

    When creating Django models, there’s no need to test the CRUD functionality since Django itself is well-tested and we can expect it to generate the correct CRUD functionality. Feel free to write some tests for practice. What really needs testing are any custom code that’s not part of Django. Sometimes we need to override the default Django behavior and that should be tested.

    Here’s a basic test to see that the model stores its name.

    app/core/tests/test_models.py#
    1def test_recurring_event(recurring_event):
2    assert str(recurring_event) == "Test Recurring Event"

    1. Pass in our fixture so that the model object is created for us.

    2. The __str__ method should be tested since it’s an override of the default Django method.

    3. Write assertion(s) to check that what’s passed into the model is what it contains. The simplest thing to check is the __str__ method.

  3. Run the test script to show it passing

    ./scripts/test.sh

1.4. Check point 1#

This is a good place to pause, check, and commit progress.

  1. Run pre-commit checks

    ./scripts/precommit-check.sh

  2. Add and commit changes

    git add -A
git commit -m "feat: add model: recurring_event"

2. Admin site#

Django comes with an admin site interface that allows admin users to view and change the data in the models. It’s essentially a database viewer.

2.1. Register the model#

  1. Import the new model

    app/core/admin.py#
    1from .models import Project

  2. Register the model with the admin site

    app/core/admin.py#
    1@admin.register(RecurringEvent)
2class RecurringEventAdmin(admin.ModelAdmin):
3    list_display = (
4        "name",
5        "start_time",
6        "duration_in_min",
7    )

    1. The class name ends with Admin to identify what it is. In this case: RecurringEventAdmin

    2. We declare a ModelAdmin class so we can customize the fields that we expose to the admin interface.

    3. We use the register decorator to register the class with the admin site.

    4. list_display controls what’s shown in the list view

    5. list_filter adds filter controls to declared fields (useful, but not shown in this example).

2.2. View the admin site#

  1. Check that everything’s working and there are no issues, which should be the case unless there’s custom input fields creating problems.

  2. See the contributing doc section on “Build and run using Docker locally” for how to view the admin interface.

  3. Example of a custom field (as opposed to the built-in ones)

    app/core/admin.py#
    1time_zone = TimeZoneField(blank=True, use_pytz=False, default="America/Los_Angeles")

    1. Having a misconfigured or buggy custom field could cause the admin site to crash and the developer will need to look at the debug message and resolve it.

2.3. Tests#

  1. Feel free to write tests for the admin. There’s no example for it yet.

  2. The reason there’s no tests is that the admin site is independent of the API functionality, and we’re mainly interested in the API part.

  3. When the time comes that we depend on the admin interface, we will need to have tests for the needed functionalities.

2.4. Check point 2#

This is a good place to pause, check, and commit progress.

  1. Run pre-commit checks

    ./scripts/precommit-check.sh

  2. Add and commit changes

    git add -A
git commit -m "feat: register admin: recurring_event"

3. API#

There’s several components to adding API endpoints: Model(already done), Serializer, View, and Router.

3.1. Add serializer#

This is code that serializes objects into strings for the API endpoints, and deserializes strings into object when we receive data from the client.

  1. Import the new model

    app/core/api/serializers.py#
    1from core.models import RecurringEvent

  2. Add a serializer class

    app/core/api/serializers.py#
     1class RecurringEventSerializer(serializers.ModelSerializer):
 2    """Used to retrieve recurring_event info"""
 3
 4    class Meta:
 5        model = RecurringEvent
 6        fields = (
 7            "uuid",
 8            "name",
 9            "start_time",
10            "duration_in_min",
11            "video_conference_url",
12            "additional_info",
13            "project",
14        )
15        read_only_fields = (
16            "uuid",
17            "created_at",
18            "updated_at",
19        )

    1. We inherit from ModelSerializer. It knows how to serialize/deserialize the Django built-in data fields so we don’t have to write the code to do it.

    2. We do need to pass in the model, the fields we want to expose through the API, and any read_only_fields.

    3. uuid, created_at, and updated_at are managed by automations and are always read-only.

  3. Custom data fields may need extra code in the serializer

    app/core/api/serializers.py#
    1time_zone = TimeZoneSerializerField(use_pytz=False)

    1. This non-built-in model field provides a serializer so we just point to it.

  4. Custom validators if we need them

    We will need to write custom validators here if we want custom behavior, such as validating URL strings and limit them to the github user profile pattern using regular expression, for example.

3.2. Add viewset#

Viewset defines the set of API endpoints for the model.

  1. Import the serializer

    app/core/api/views.py#
    1from ..models import RecurringEvent

  2. Add the viewset and CRUD API endpoint descriptions

    app/core/api/views.py#
     1@extend_schema_view(
 2    list=extend_schema(description="Return a list of all the recurring events"),
 3    create=extend_schema(description="Create a new recurring event"),
 4    retrieve=extend_schema(description="Return the details of a recurring event"),
 5    destroy=extend_schema(description="Delete a recurring event"),
 6    update=extend_schema(description="Update a recurring event"),
 7    partial_update=extend_schema(description="Patch a recurring event"),
 8)
 9class RecurringEventViewSet(viewsets.ModelViewSet):
10    permission_classes = [IsAuthenticated]
11    queryset = RecurringEvent.objects.all()
12    serializer_class = RecurringEventSerializer

    1. We inherit from ModelViewSet, which provides a default view implementation of all 6 CRUD actions: create, retrieve, partial_update, update, destroy, list.

    2. We use the extend_schema_view decorator to attach the API doc strings to the viewset. They are usually defined as docstrings of the corresponding function definitions inside the viewset. Since we use ModelViewSet, there’s nowhere to put the docstrings but above the viewset.

    3. The minimum code we need with ModelViewSet are the queryset, and the serializer_class.

    4. Permissions

      1. For now use permission_classes = [IsAuthenticated]

      2. It doesn’t control permissions the way we want, but we will fix it later.

3.3. Extended example#

Query params

This example shows how to add a filter params. It’s done for the user model as a requirement from VRMS.

app/core/api/views.py#
 1@extend_schema_view(
 2    list=extend_schema(
 3        summary="Users List",
 4        description="Return a list of all the existing users",
 5        parameters=[
 6            OpenApiParameter(
 7                name="email",
 8                type=str,
 9                description="Filter by email address",
10                examples=[
11                    OpenApiExample(
12                        "Example 1",
13                        summary="Demo email",
14                        description="get the demo user",
15                        value="demo-email@email.com,",
16                    ),
17                ],
18            ),
19            OpenApiParameter(
20                name="username",
21                type=OpenApiTypes.STR,
22                location=OpenApiParameter.QUERY,
23                description="Filter by username",
24                examples=[
25                    OpenApiExample(
26                        "Example 1",
27                        summary="Demo username",
28                        description="get the demo user",
29                        value="demo-user",
30                    ),
31                ],
32            ),
33        ],
34    ),
35    create=extend_schema(description="Create a new user"),
36    retrieve=extend_schema(description="Return the given user"),
37    destroy=extend_schema(description="Delete the given user"),
38    update=extend_schema(description="Update the given user"),
39    partial_update=extend_schema(description="Partially update the given user"),
40)
41class UserViewSet(viewsets.ModelViewSet):
42    permission_classes = [IsAuthenticated]
43    serializer_class = UserSerializer
44    lookup_field = "uuid"
45
46    def get_queryset(self):
47        """
48        Optionally filter users by an 'email' and/or 'username' query paramerter in the URL
49        """
50        queryset = get_user_model().objects.all()
51        email = self.request.query_params.get("email")
52        if email is not None:
53            queryset = queryset.filter(email=email)
54        username = self.request.query_params.get("username")
55        if username is not None:
56            queryset = queryset.filter(username=username)
57        return queryset

  1. Write API documentation

    1. Define strings for all 6 actions: create, retrieve, partial_update, update, destroy, list.

    2. This one is fancy and provides examples of data to pass into the query params. It’s probably more than we need right now.

      1. The examples array can hold multiple examples.

        1. Example ID string has to be unique but is not displayed.

        2. summary string appears as an option in the dropdown.

        3. description is displayed in the example.

  2. Add query params

    1. Notice the queryset property is now the get_queryset(() function which returns the queryset.

    2. The get_queryset() function overrides the default and lets us filter the objects returned to the client if they pass in a query param.

    3. Start with all the model objects and filter them based on any available query params.

3.4. Register API endpoints to the router#

  1. Import the viewset.

    app/core/api/urls.py#
    1from .views import RecurringEventViewSet

  2. Register the viewset to the router

    app/core/api/urls.py#
    1router.register(r"recurring-events", RecurringEventViewSet, basename="recurring-event")

    1. First param is the URL prefix use in the API routes. It is, by convention, plural

      • This would show up in the URL like this: http://localhost/api/v1/recuring-events/ and http://localhost/api/v1/recuring-events/<uuid>

    2. Second param is the viewset class which defines the API actions

    3. basename is the name used for generating the endpoint names, such as [basename]-list, [basename]-detail, etc. It’s in the singular form. This is automatically generated if the viewset definition contains a queryset attribute, but it’s required if the viewset overrides that with the get_queryset function

      • reverse("recurring-event-list") would return http://localhost/api/v1/recuring-events/

3.5. Add API tests#

For the CRUD operations, since we’re using ModelViewSet where all the actions are provided by rest_framework and well-tested, it’s not necessary to have test cases for them. But here’s an example of one.

  1. Import API URL

    app/core/tests/test_api.py#
    1RECURRING_EVENTS_URL = reverse("recurring-event-list")

  2. Add test case

    app/core/tests/test_api.py#
     1def test_create_recurring_event(auth_client, project):
 2    """Test that we can create a recurring event"""
 3
 4    payload = {
 5        "name": "Test Weekly team meeting",
 6        "start_time": "18:00:00",
 7        "duration_in_min": 60,
 8        "video_conference_url": "https://zoom.com/link",
 9        "additional_info": "Test description",
10        "project": project.uuid,
11    }
12    res = auth_client.post(RECURRING_EVENTS_URL, payload)
13    assert res.status_code == status.HTTP_201_CREATED
14    assert res.data["name"] == payload["name"]

    1. Given

      1. Pass in the necessary fixtures

      2. Construct the payload

    2. When

      1. Create the object

    3. Then

      1. Check that it’s created via status code

      2. Maybe also check the data. A real test should check all the data, but we’re kind of relying on django to have already tested this.

  3. Run the test script to show it passing

    ./scripts/test.sh

3.6. Check point 3#

This is a good place to pause, check, and commit progress.

  1. Run pre-commit checks

    ./scripts/precommit-check.sh

  2. Add and commit changes

    git add -A
git commit -m "feat: add endpoints: recurring_event"

3.7. Push the code and start a PR#

Refer to the contributing doc section on “Push to upstream origin” onward.