Chapter 6: Working with Forms and Validators
++++++++++++++++++++++++++++++++++++++++++++
.. index ::
single: Chap 6-Working with Forms and Validators
Form handling is one of those areas that at first glance appears very simple but in real applications can quickly become rather complicated. There are generally two approaches to dealing with forms. The first is to code all your forms, validation, and logic manually to give you complete control over how your forms work. The alternative approach is to use a form framework where ready-made classes exist for each of the field types you might want to use. The form framework then automates the generation of HTML, the validation of data, and the display of error messages for you.
At first glance, it might appear that a form framework would save you a lot of time, but in reality, form frameworks are rarely flexible enough to deal with all the situations you might want to develop, and in the long run you can sometimes find yourself spending more time creating custom fields for your form framework than it would have taken if you had coded all your forms manually.
.. index ::
single: for form handling; tools
Because of this, Pylons encourages you to do a lot of the work of generating forms yourself, but it does provide four sets of tools to make form handling as painless as possible:
* Form helpers to generate the HTML for common field types
* Validators to validate form data and convert between HTML and Python representations of particular data types
* HTML Fill to take an HTML form and automatically populate it with values and error messages for redisplaying the form data
* The ``@validate`` decorator to automate the process of validating a form and redisplaying it if it contains invalid data
.. index ::
single: ToscaWidgets; tools
single: approaches to; forms
These four tools can help make handling forms much simpler without in any way constraining your creativity as a developer. Pylons does support an alternative approach with a tool, called ToscaWidgets, although it won’t be covered in this chapter. ToscaWidgets is a full form framework developed from the original widgets code in TurboGears that automates every aspect of form handling. ToscaWidgets is still officially in prerelease, but if you are interested in its approach, you should visit http://toscawidgets.org to find out more. The majority of developers prefer the flexibility of the approach you’ll use in this chapter.
The Basics
==========
.. index ::
single: overview of; forms
When a user submits a form on a web site, the data is submitted to the URL specified in the ``action`` attribute of the ``
If your form contains a file upload field such as ````, you will also need to specify an ``enctype="multipart/form-data"`` attribute, and you have to choose the ``post`` method.
Many people put the value of the ``method`` attribute in uppercase. If your HTML page uses XHTML, the ``method`` attribute value is supposed to be lowercase, which is why in this example it is specified as ``method="get"``, not ``method="GET"``, as many examples will show.
Later in the chapter, you’ll see how you can improve this example by using the ``h.url_for()`` helper in the form action and by using Pylons’ field helpers to generate most of the HTML for the form automatically. First, though, let’s create a new Pylons project to test this example as it stands:
::
$ paster create --template=pylons FormDemo
Accept the default options by pressing Enter to choose Mako as the template engine and no SQLAlchemy or Google App Engine support.
.. index ::
single: simple template for; forms
Once the project has been created, let’s create a simple template in ``FormDemo/formdemo/templates/base.html`` to use as a basis for the examples in this chapter:
::
FormDemo
${next.body()}
Create a new template called ``simpleform.html`` with the following content to test the example form:
::
<%inherit file="/base.html" />
Enter Your Email Address
.. index ::
single: %(%inherit%) tag
You’ll remember from the previous chapter that the ``<%inherit>`` tag allows the body of a template to be inserted into a parent template.
.. index ::
single: controllers for; forms
Now create a new controller called ``formtest``:
::
$ cd FormDemo
$ paster controller formtest
Add two actions to the controller that look like this:
::
def form(self):
return render('/simpleform.html')
def submit(self):
return 'Your email is: %s' % request.params['email']
Start the server:
::
$ paster serve --reload development.ini
Visit http://localhost:5000/formtest/form, and you will see the form. In this case, the generated HTML looks like this:
::
FormDemo
Enter Your Email Address
Try entering the e-mail address ``test@example.com`` and clicking Submit. The URL should change to http://localhost:5000/formtest/submit?email=test%40example.com&submit=Submit, and you should see the text ``Your email is: test@example.com``.
.. index ::
single: request.params object and; forms
single: getall() method
single: get(key, default) on request.params object
Pylons has parsed and decoded the query string and set up the ``request.params`` object you saw in Chapter 3. As you’ll recall, this object behaves a bit like a dictionary where the keys are the names of the fields in the form, and their corresponding values are Unicode strings, with all the characters in the query string properly decoded ready for you to use. If you have two fields with the same name in the form, then using the dictionary interface will return the first string. You can get all the strings returned as a list by using the ``.getall()`` method. If you expect only one value and want to enforce this, you should use ``.getone()``, which raises an error if more than one value with the same name is submitted. By default, if a field is submitted without a value, the dictionary interface returns an empty string. This means that using ``.get(key, default)`` on ``request.params`` will return a default only if the value was not present in the form.
POST vs. GET
------------
.. index ::
single: submitting using GET or POST HTTP methods; forms
single: method attribute; %(form%) tag
single: GET and POST requests in; LiveHTTPHeaders
Forms can be submitted using either GET or POST HTTP methods depending on the value you set for the ``method`` attribute of the ``
Error Message Formatting
------------------------
.. index ::
single: error message formatting; HTML Fill tool
The error message formatting might not be quite what you were after, so HTML Fill defines two special tags that can be used to customize how the error messages are displayed:
````:
This tag is eliminated completely if there is no error for the named field. Otherwise, the error is passed through the given formatter (``default()`` if no ``format`` attribute is given).
``...``:
If the named field doesn’t have an error, everything between the tags will be eliminated. Use ``name="not field_name"`` to invert the behavior (in other words, include text only if there are no errors for the field).
.. index ::
single: formatters
Formatters are functions that take the error text as a single argument and return a string that is inserted into the template. Formatters are specified as arguments to the ``htmlfill.render()`` function, which I will describe next. The default formatter returns the following:
::
(message)
where ``(message)`` is replaced with the error message concerned. Most of the time it is best to use a formatter because the second form displays the static HTML you’ve specified, not the actual error message generated.
If any errors are generated for fields that don’t exist, they are added at the top of the form.
Render Arguments
----------------
.. index ::
single: render() function; HTML Fill tool
HTML Fill’s ``render()`` function has the following arguments that you can use to customize how the form is rendered:
::
def render(form, defaults=None, errors=None, use_all_keys=False,
error_formatters=None, add_attributes=None,
auto_insert_errors=True, auto_error_formatter=None,
text_as_default=False, listener=None)
It is important to note that HTML Fill’s ``render()`` function has nothing to do with the ``render()`` function you’ve been using to render templates; it is just unfortunate that both have the same name.
The example so far has used the ``form``, ``defaults`` and ``errors`` arguments, but other options can be useful too:
``use_all_keys``
If this is ``True``, if there are any extra fields from defaults or errors that couldn’t be used in the form, it will be an error.
``error_formatters``
This is a dictionary of formatter names to one-argument functions that format an error into HTML. Some default formatters are provided if you don’t provide this.
``add_attributes``
This is a dictionary of field names to a dictionary of attribute name/values. If the name starts with ``+``, then the value will be appended to any existing attribute (for example, ``{'+class': ' important'}``).
``auto_insert_errors``
If this is ``True`` (the default), then any errors for which ```` tags can’t be found will be put just above the associated input field, or at the top of the form if no field can be found.
``auto_error_formatter``
This is used to create the HTML that goes above the fields. By default, it wraps the error message in a span and adds a `` ``.
``text_as_default``
If this is ``True`` (the default is ``False``), then ```` will be treated as text inputs.
.. index ::
single: render() function; HTML Fill tool
``listener``
This can be an object that watches fields pass; the only one currently is in ``formencode.htmlfill_schemabuilder.SchemaBuilder``.
Doing Validation the Quick Way
==============================
.. index ::
single: validate() decorator
Now that you’ve seen in detail how to use FormEncode and HTML Fill, you’ll be pleased to know that Pylons provides an even simpler way of using the same functionality that is suitable for the majority of use cases you are likely to encounter.
You can use it like this:
::
def form(self):
return render('/simpleform.html')
@validate(schema=EmailForm(), form='form', post_only=False, on_get=True)
def submit(self):
return 'Your email is: %s and the date selected was %r.' % (
self.form_result['email'],
self.form_result['date'],
)
.. index ::
single: validate() decorator; importing
You’ll need to import the ``@validate`` decorator at the top of the controller:
::
from pylons.decorators import validate
What this says is that if the data submitted to the ``submit()`` action contains any errors, then the request should be rerun as a GET request to the ``form()`` action. The result of calling the ``form()`` action is then passed through HTML Fill to render the errors and repopulate the form with the values that were submitted.
.. note ::
Python 2.3 doesn’t support decorators, so rather than using the ``@validate()`` syntax, you need to put ``email = validate(schema=EmailForm(), form='form', post_only=False, on_get=True)(email)`` after the e-mail function’s declaration.
.. index ::
single: validate() decorator
By default, if you don’t specify ``post_only=False`` and ``on_get=True`` to the ``@validate`` decorator, validation would occur only on POST requests, so you would need to alter your form definition so that the method is a POST:
::
<% h.form(h.url_for(action='submit'), method='post') %>
.. caution ::
If you do this, calling an action wrapped by ``@validate`` using a GET request will bypass the validation and call the action anyway. You need to make sure this doesn’t pose a security risk in your application. You could prevent this by testing whether a GET or a POST is being used in the body of the action. The request method can be determined using ``request.method``.
.. index ::
single: customizing call for; HTML Fill tool
You can customize the way HTML Fill is called by passing any of the arguments accepted by ``htmlfill.render()`` as keyword arguments to ``validate()``. For example, to specify a custom error formatter, you can do this:
::
from formencode import htmlfill
def custom_formatter(error):
return '%s \n' % (
htmlfill.html_quote(error)
)
Update the ``submit()`` action to look like this:
::
@validate(schema=EmailForm(), form='form', post_only=False, on_get=True,
auto_error_formatter=custom_formatter)
def submit(self):
return 'Your email is: %s and the date selected was %r.' % (
self.form_result['email'],
self.form_result['date'],
)
With this in place, the error messages will be wrapped in ```` tags. If you run the example, you will notice that the error messages are no longer highlighted in red because there is no style set up for the new error class.
You can pass other options to the HTML Fill ``render()`` function in the same way via the ``@validate`` decorator.
.. index ::
single: validate() decorator
The ``@validate`` decorator is documented at http://docs.pylonshq.com/modules/decorators#module-pylons.decorators.
Using Custom Validators
=======================
.. index ::
single: custom, creating; FormEncode tool
FormEncode comes with a useful set of validators, but you can also easily create your own. One common reason for wanting to do this is if you are populating a select field with values from a database and you want to ensure the value submitted is one of the values in the database.
For this example, imagine you have a function called ``get_option_values()`` that interacts with a database every time it is called and returns a list of valid integers.
Here’s a potential implementation:
::
class ValidOption(formencode.validators.FancyValidator):
def validate_python(self, value, state):
valid_values = get_option_values()
if value not in valid_values:
raise formencode.Invalid("Invalid value", value, state)
When the validator is used in a schema and checked, an error message will be displayed if the value submitted isn’t one of the options returned by ``get_option_values()``.
You might use it like this:
::
class EmailForm(formencode.Schema):
allow_extra_fields = True
filter_extra_fields = True
email = formencode.validators.Email(not_empty=True)
date = formencode.validators.DateConverter(not_empty=True)
option = ValidOption()
.. index ::
single: validate_python() method
Let’s have a look at the implementation in more detail. Notice that the ``validate_python()`` method also takes a ``state`` argument. It’s used for very little in the validation system but provides a mechanism for passing information from Pylons to any custom validators you write. The ``c`` global is used in Pylons for storing per-request state information, so it makes sense to also use it as the state argument to your validators, although you are free to use other objects if you prefer.
.. index ::
single: custom, creating; FormEncode tool
As an example, let’s imagine that ``get_option_values()`` relied on a database connection that was set up on each request. You couldn’t pass the connection as an argument to ``ValidOption`` because the connection wouldn’t exist at the point Python created the schema. The connection exists only during the request, so you would attach it to the ``c`` global during the request and pass the ``c`` global as the ``state`` argument when instantiating the schema. Here’s how the start of the ``submit()`` action might look from the full FormEncode and HTML Fill example earlier:
::
def submit(self):
# Imagine we have a connection object now:
c.connection = connection
schema = EmailForm(state=c)
... same as before
You could now update the example like this:
::
class ValidOption(formencode.validators.FancyValidator):
def validate_python(self, value, c):
valid_values = get_option_values(c.connection)
if value not in valid_values:
raise formencode.Invalid("Invalid value", value, c)
In this way, the validator can use request-specific information even though it is defined before the request starts.
.. index ::
single: validate_python() method
The previous implementation uses the ``validate_python()`` method, which simply raises an exception if it needs to do so. This is useful in this example because the conversion needs to be done by the validator when converting to and from Python. This won’t be the case for all the validators you create, though. The ``_to_python()`` and ``_from_python()`` methods are provided for you to implement any conversion code to convert to or from Python, respectively.
The ``validate_python()`` method is called *after* ``_to_python()`` so that the ``value`` argument will be a normal Python object by the time it comes to being validated. ``validate_python()`` is called *before* ``_from_python()``. Both ``_to_python()`` and ``_from_python()`` take the same arguments as ``validate_python()``.
.. index ::
single: custom, creating; FormEncode tool
You’ll remember from earlier in the chapter that validators can be customized when you instantiate them in a schema. At the moment, the validator you’ve created always displays the message ``Invalid value``. Let’s update the validator to allow it to be customized:
::
class ValidOption(formencode.validators.FancyValidator):
messages = {
'invalid': 'Invalid value',
}
def validate_python(self, value, c):
valid_values = get_option_values(c.connection)
if value not in valid_values:
raise formencode.Invalid(
self.message("invalid", c),
value,
c
)
You can also include values in the message itself like this:
::
class ValidOption(formencode.validators.FancyValidator):
messages = {
'invalid': 'Invalid value %(invalid)s',
}
def validate_python(self, value, c):
valid_values = get_option_values(c.connection)
if value not in valid_values:
raise formencode.Invalid(
self.message("invalid", c, invalid=value),
value,
c
)
.. index ::
single: custom, creating; FormEncode tool
The message string specified gets interpolated with a dictionary made from the keyword arguments you pass to the ``self.message()`` function. This system is designed to make the messages easy to format for different environments or replaceable for different languages.
.. index ::
single: Invalid exception class
You’ll also notice that you use a special exception class, ``formencode.Invalid``, to raise an error. This is the same exception class you catch in the controller action and use to obtain the values to pass to ``htmlfill.render()``. Besides the string error message, ``Invalid`` exceptions have a few other instance variables:
``value``
This is the input to the validator that failed.
``state``
This is the associated state.
``msg``
This is the error message (``str(error)`` returns this).
``error_list``
If the exception happened in a ``ForEach`` (list) validator, then this will contain a list of ``Invalid`` exceptions. Each item from the list will have an entry, either ``None`` for no error or an exception.
``error_dict``
If the exception happened in a ``Schema`` (dictionary) validator, then this will contain ``Invalid`` exceptions for each failing field.
``.unpack_errors()``
This method returns a set of lists and dictionaries containing strings for each error. It’s an unpacking of ``error_list``, ``error_dict``, and ``msg``. If you get an ``Invalid`` exception from a ``Schema``, you probably want to call this method on the exception object.
.. index ::
single: FancyValidator class
single: custom, creating; FormEncode tool
If you are interested in writing your own validators, it is useful to see the source code for the ``FancyValidator`` class. It is in the ``formencode.api`` module and explains all the options and methods validators have. There are other types of validators too such as compound validators and chained validators. Generally speaking, the best way to implement your own validator is to look at the source code of an existing validator that behaves in a similar manner and implement your own validator in the same way.
Solving the Repeating Fields Problem
====================================
.. index ::
single: overview of; forms
In real-world examples, you rarely just need a form that can be populated from a flat dictionary structure. Forms that contain subforms or repeating sets of fields are actually very common.
Let’s imagine a situation where you are writing a form to allow a researcher to add information about a research project they are conducting. They might need to provide the following information:
* The title of the study
* When it is going to start and end
* The contact details of the people who are participating
Contact details consist of the following information:
* Title
* First name
* Surname
* Role in the project
Let’s also imagine that a research project requires at least one person to be added and also that there can be only one person with the role of chief investigator.
You can easily provide a form to enter the study title, start date, and end date, but providing a form to allow an unknown number of people’s contact details to be entered is slightly trickier. You can take one of two approaches:
* Design the form in a wizard format. The user enters the title, start date, and end date on the first screen and clicks Submit. The data is saved, and the second screen is displayed, allowing the user to add the contact details of the first person. Once they have submitted the form, they are asked whether they want to add another person. This continues until they have added all the necessary data.
* Display a single form containing the title, start date, and end date as before but with a button to allow the user to add fields to add a person. When the user clicks the Add New Person button, a set of fields to enter the first person are shown. The user can submit the form or click the Add New Person button again to add another set of fields. Finally, once they have completed the whole form containing the study and person data, they click Save, and the data is validated and saved in one go.
.. index ::
single: solutions to; forms
The advantage of the first approach is that at any one time you are dealing with only one set of fields that can be submitted as simple name/value pairs, so the data structure is very straightforward. The disadvantage is that at each step in the wizard you need to store the data that has already been submitted, and this has its own problems. Imagine, for example, you save the study information in the first step but the user changes their mind and never completes the second step of adding a person. One of the requirements was that studies should have at least one person associated with them, but now you have saved a study that doesn’t have any people. Say, instead, that the user does add a person and gives them a role of chief investigator. Now let’s imagine they add another person and give them a role of chief investigator too. You can’t have more than one chief investigator, so the validation code will display an error explaining the problem. Imagine, though, that the user really does want the second person to be the chief investigator and made a mistake in giving the first person the chief investigator role. The user has no choice but to start the form again.
Obviously, all the problems with the wizard approach can be solved. You can store the data in a temporary location or a session store and save it properly only at the end of the wizard, or you can provide Back buttons so the user can go back through the wizard and make changes, but it can be a surprising amount of work to program the logic and validation code for this sort of workflow.
.. index ::
single: solutions to; forms
The major advantage of the second approach is that all the required data is stored client-side throughout the submission and validation cycles, which means your Pylons controller needs to store the data only once, after all the input has been validated. This can greatly reduce the complexity of your controller logic. FormEncode provides the necessary tools to help you with this.
Creating the Form
-----------------
.. index ::
single: schema for; forms
The first thing you need to do is define the schema. Here’s what the main study schema might look like:
::
class Study(Schema)
title = String(not_empty=True)
start_date = DateConverter()
end_date = DateConverter()
people = ForEach(Person())
.. index ::
single: ForEach validator
As you can see, the schema has ``title``, ``start_date``, and ``end_date`` fields, which use ordinary validators as you would expect, but there is also a ``people`` field that takes a ``ForEach()`` validator. The ``ForEach()`` validator takes a single argument, which is another validator or schema it should validate. In this case, you want it to validate people, so you’ve specified an instance of a ``Person`` schema. The ``Person`` schema looks like this:
::
class Person(Schema):
title = String()
firstname = String(not_empty=True)
surname = String(not_empty=True)
role = OneOf(['1', '2', '3'])
.. index ::
single: role field values; forms
The ``role`` field will be a select drop-down that takes the values in Table 6-2.
==================== =======
Name Value
==================== =======
Chief investigator 1
Assistant 2
Student 3
==================== =======
Table 6-2. Possible Values for Role
.. index ::
single: OneOf validator
The ``OneOf`` validator checks that the value submitted for the ``role`` is one of the values specified.
.. index ::
single: schema for; forms
Now turn your attention to the template defs to produce the fields. You’ll create a working example as you go through this chapter, so let’s create a project for it (accept the default values when prompted):
::
$ paster create --template=pylons FormExample
$ cd FormExample
$ paster controller study
Create two directories named ``base`` and ``derived`` in the ``templates`` directory, and add a ``base/index.html`` file that looks like this:
::
FormsExample
${next.body()}
.. index ::
single: derived/form.html template; listings
Then create a new template called ``derived/form.html`` with the following content:
::
<%inherit file="/base/index.html" />
<%def name="study()">
% for id in range(c.number_of_people):
${person(id=id)}
% endfor
<%def name="person(id)">
Create a Study
${h.form(h.url_for(controller='study', action='process'))}
${study()}
${h.submit(name="action", value="Save")}
${h.submit(name="action", value="Add New Person")}
${h.end_form()}
.. index ::
single: schema for; forms
Rendering this template would result in the ``study()`` def being called, and this in turn would call the ``person()`` def to create a set of fields for the number of people specified in ``c.number_of_people``, which you will set in the controller in a minute.
.. tip ::
The template you’ve created uses ``