Hi Sean,
I had a discussion about a month ago with R&D when someone asked what fields could be searched. The list of fields is in the code and not user configurable
“You can search on the activity only according to the fields that are on the sorting list.”
They would have to add this as a change request if it were possible but I am not sure the address is something that is very feasible for sorting so may not be a candidate to add.
Perhaps to help a bit, maybe you can set up some filters which would not be practical with address fields but perhaps cities or states? I understand there may still be too many permutations but just checking.
Otherwise, you would need to ask for a customization change request to implement this if it were possible.