CS534 — Written Assignment 4 — Due Nov 18th 1. L2 SVM Given a set of training examples {(xi , yi )}N i=1 , where yi ∈ {1, −1} for all i. The following is the primal formulation of L2 SVM, a variant of the standard SVM obtained by squaring the hinge loss: min wT w + λ w,b,ξ N X ξi2 i=1 s.t. yi (wT xi + b) ≥ 1 − ξi , i ∈ {1, · · · , N } xi ≥ 0, i ∈ {1, · · · , N } a. Show that removing the second constraint ξi ≥ 0 will not change the solution to the primal problem. In other words, let (w∗ , b∗ , ξ ∗ ) be the optimal solution to the problem without this set of constraints, show that ξi∗ ≥ 0. b. After removing the second set of constraints, we have a simpler problem with only one set of constraints. Now provide the lagrangian of this new problem. c. Derive the dual of this problem. How is it different from the standard SVM with hinge loss? 2. Neural network expressiveness In class, we have discussed that neural network can express any arbitrary boolean functions. Please answer the following question about neural networks. a. It is impossible to implement a XOR function y = x1 ⊕ x2 using a single unit (neuron). However, you can do it with a neural net. Use the smallest network you can. Draw your network and show all the weights. b. Create a neural network with only one hidden layer that implements (x1 ∨ ¬x2 ) ⊕ (¬x3 ∨ ¬x4 ). c. Explain how can we construct a neural network to implement a Naive Bayes Classifier with Boolean features. d. Explain how can we construct a neural network to implement a decision tree classifier with boolean features. 3. PAC learnability. Consider the concept class C of all conjunctions (allowing negations) over n boolean features. Prove that this concept class is PAC learnable. Note that we have gone through this example in class but not in full detail. For this problem, you should work out the full detail, including showing that there exists a poly-time algorithm for finding a consistent hypothesis. 4. Consider the class C of concepts of the form (a ≤ x ≤ b) ∧ (c ≤ y ≤ d), where a, b, c, and d are integers in the interval [0, 99]. Note that each concept in this class corresponds to a rectangle with integer-valued boundaries on a portion of the (x, y) plane. Hint: Given a region in the plane bounded by the points (0, 0) and (n − 1, n − 1), the number of distinct rectangles with integer-valued boundaries 2 n(n − 1) . within this region is 2 (a) Give an upper bound on the number of randomly drawn training examples sufficient to assure that for any target concept c in C, any consistent learner using H = C will, with probability 95%, output a hypothesis with error at most 0.15. (b) Now suppose the rectangle boundaries a, b, c, and d take on real values instead of integer values. Update your answer to the first part of this question. 1